2021-05-12T11:42:14.189 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-12T11:42:14.190 DEBUG:teuthology.run:Teuthology command: teuthology --name kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi --archive /home/teuthworker/archive/kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi/6111450 --verbose --description rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/osd} --owner scheduled_kchai@teuthology -- /home/teuthworker/archive/kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi/6111450/orig.config.yaml 2021-05-12T11:42:14.221 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-05-12T11:42:14.297 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi/6111450 branch: wip-kefu-testing-2021-05-12-1746 description: rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/osd} email: null first_in_suite: false job_id: '6111450' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: rhel os_version: '8.3' overrides: admin_socket: branch: wip-kefu-testing-2021-05-12-1746 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 36c2cf2fb22a9790b2c097e158985773b35f1740 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 36c2cf2fb22a9790b2c097e158985773b35f1740 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-kefu-testing-2021-05-05-1449 sha1: 5cd0ad0001da189274219f5e829752717eb856d2 owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 36c2cf2fb22a9790b2c097e158985773b35f1740 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-05-05-1449 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5cd0ad0001da189274219f5e829752717eb856d2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5cd0ad0001da189274219f5e829752717eb856d2 targets: smithi191.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMI8eCv4lcqYBL8TV5FzhbvG5GF3fgmgSqFIHIAn72bkBrVk5RjYJtrCanwtnkExsE2a7aIFpdurOyQPZjsHm+yzISwX7pDaLWeThRXyGOPckUEO1Qg8fuKWH8WS/u09p5hiXoac6jxav2Vi54OOYKmypMSg/ZJpCGVzHVoECxd6/+89g6SGeV51P0s3D/aa22zbBkdY3Om5UR8gm8gkDjl9R+XP/B2ay3BE4gEfldPcaS1ZNeLgINUfwr8Jx8FjGzpNMZxHIASGX+W0aR+MU8vEOTBz6MYswb6llXLUV4G6AaN5z2uVWvb1TQXoOezbJpdUBpDIgqZk+PhpIwTRMktsb2mUdFoAM3xUQL7rF7XvhH55BXWdmBBe/woWIDNORaNbEC1TX32RDxozjrjwWo6mx5qm5/6/Ca7pEkQP7NOhw6CVW0qBMb/7Jcl2daumBu0lg1U+ljp2FFO0xMKJ7bsrEK6XYTh33+4GhG+yBVFVTNop9CHH5SYFEdrs++guE= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master teuthology_sha1: 19220a3bd6e252c6e8260827019668a766d85490 timestamp: 2021-05-12_11:35:51 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.8274 2021-05-12T11:42:14.297 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5cd0ad0001da189274219f5e829752717eb856d2/qa; will attempt to use it 2021-05-12T11:42:14.298 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5cd0ad0001da189274219f5e829752717eb856d2/qa/tasks 2021-05-12T11:42:14.298 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-05-12T11:42:14.301 INFO:teuthology.task.internal:Checking packages... 2021-05-12T11:42:14.322 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '36c2cf2fb22a9790b2c097e158985773b35f1740' 2021-05-12T11:42:14.322 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-05-12T11:42:14.323 INFO:teuthology.packaging:ref: None 2021-05-12T11:42:14.323 INFO:teuthology.packaging:tag: None 2021-05-12T11:42:14.323 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-05-12-1746 2021-05-12T11:42:14.324 INFO:teuthology.packaging:sha1: 36c2cf2fb22a9790b2c097e158985773b35f1740 2021-05-12T11:42:14.324 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2021-05-12-1746 2021-05-12T11:42:14.473 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-4087.g36c2cf2f 2021-05-12T11:42:14.474 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-05-12T11:42:14.485 INFO:teuthology.task.internal:no buildpackages task found 2021-05-12T11:42:14.485 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-05-12T11:42:14.522 INFO:teuthology.task.internal:Saving configuration 2021-05-12T11:42:14.534 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-05-12T11:42:14.555 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-12T11:42:14.571 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-12 11:37:12.481041', 'description': '/home/teuthworker/archive/kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi/6111450', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECJMClLmnozKKLTab0Q8s3oJWbhmF8gWImEcDJOgJIfC/a/S5kR5QYNTBxBMHyjhznNoGnXUjEnkTcVvIW35x0=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi191.front.sepia.ceph.com'} 2021-05-12T11:42:14.572 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-05-12T11:42:14.585 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-05-12T11:42:14.585 INFO:teuthology.run_tasks:Running task console_log... 2021-05-12T11:42:14.641 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbf4dad9d90>, signals=[15]) 2021-05-12T11:42:14.643 INFO:teuthology.run_tasks:Running task internal.connect... 2021-05-12T11:42:14.653 INFO:teuthology.task.internal:Opening connections... 2021-05-12T11:42:14.653 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2021-05-12T11:42:14.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:42:14.728 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-05-12T11:42:14.743 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2021-05-12T11:42:14.893 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2021-05-12T11:42:14.894 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2021-05-12T11:42:14.948 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="Red Hat Enterprise Linux" 2021-05-12T11:42:14.949 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="8.3 (Ootpa)" 2021-05-12T11:42:14.949 INFO:teuthology.orchestra.run.smithi191.stdout:ID="rhel" 2021-05-12T11:42:14.949 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="fedora" 2021-05-12T11:42:14.949 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="8.3" 2021-05-12T11:42:14.950 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el8" 2021-05-12T11:42:14.950 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-05-12T11:42:14.950 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2021-05-12T11:42:14.950 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-05-12T11:42:14.951 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://www.redhat.com/" 2021-05-12T11:42:14.951 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-05-12T11:42:14.951 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:42:14.951 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-05-12T11:42:14.952 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-05-12T11:42:14.952 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-05-12T11:42:14.952 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-05-12T11:42:14.953 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2021-05-12T11:42:14.971 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-05-12T11:42:14.983 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-05-12T11:42:15.006 INFO:teuthology.task.internal:Checking for old test directory... 2021-05-12T11:42:15.007 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2021-05-12T11:42:15.021 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-05-12T11:42:15.034 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-05-12T11:42:15.034 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2021-05-12T11:42:15.109 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-05-12T11:42:15.136 INFO:teuthology.run_tasks:Running task kernel... 2021-05-12T11:42:15.157 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-05-12T11:42:15.157 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-05-12T11:42:15.158 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2021-05-12T11:42:15.172 INFO:teuthology.orchestra.run.smithi191.stdout:4.18.0-240.1.1.el8_3.x86_64 2021-05-12T11:42:15.172 INFO:teuthology.task.kernel:Running kernel on smithi191: 4.18.0-240.1.1.el8_3.x86_64 2021-05-12T11:42:15.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2021-05-12T11:42:16.749 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:42:16.752 INFO:teuthology.orchestra.run.smithi191.stdout:Unable to read consumer identity 2021-05-12T11:42:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:42:16.862 INFO:teuthology.orchestra.run.smithi191.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-05-12T11:42:16.862 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:42:19.372 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 9.4 MB 00:01 2021-05-12T11:42:21.280 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 5.2 MB/s | 24 kB 00:00 2021-05-12T11:42:21.878 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-4.18.0-240.1.1.el8_3.x86_64 is already installed. 2021-05-12T11:42:21.878 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-05-12T11:42:21.998 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:42:22.011 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:42:22.011 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:42:22.119 DEBUG:teuthology.orchestra.run.smithi191:> echo no | sudo yum reinstall kernel || true 2021-05-12T11:42:22.380 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:42:22.380 INFO:teuthology.orchestra.run.smithi191.stdout:Unable to read consumer identity 2021-05-12T11:42:22.428 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:42:22.428 INFO:teuthology.orchestra.run.smithi191.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-05-12T11:42:22.428 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:42:22.601 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:01 ago on Wed 12 May 2021 11:42:21 AM UTC. 2021-05-12T11:42:22.724 INFO:teuthology.orchestra.run.smithi191.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-05-12T11:42:22.724 INFO:teuthology.orchestra.run.smithi191.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-05-12T11:42:22.767 INFO:teuthology.orchestra.run.smithi191.stderr:Error: No packages marked for reinstall. 2021-05-12T11:42:22.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum reinstall -y kernel || true 2021-05-12T11:42:23.064 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:42:23.064 INFO:teuthology.orchestra.run.smithi191.stdout:Unable to read consumer identity 2021-05-12T11:42:23.107 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:42:23.107 INFO:teuthology.orchestra.run.smithi191.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-05-12T11:42:23.108 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:42:23.279 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:02 ago on Wed 12 May 2021 11:42:21 AM UTC. 2021-05-12T11:42:23.407 INFO:teuthology.orchestra.run.smithi191.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-05-12T11:42:23.407 INFO:teuthology.orchestra.run.smithi191.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-05-12T11:42:23.437 INFO:teuthology.orchestra.run.smithi191.stderr:Error: No packages marked for reinstall. 2021-05-12T11:42:23.473 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel --last | head -n 1 2021-05-12T11:42:23.570 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Tue 10 Nov 2020 09:16:49 PM UTC 2021-05-12T11:42:23.571 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2021-05-12T11:42:23.571 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-05-12T11:42:23.572 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-05-12T11:42:23.572 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-05-12T11:42:23.601 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2021-05-12T11:42:23.616 INFO:teuthology.run_tasks:Running task internal.base... 2021-05-12T11:42:23.628 INFO:teuthology.task.internal:Creating test directory... 2021-05-12T11:42:23.628 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-05-12T11:42:23.670 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-05-12T11:42:23.680 INFO:teuthology.run_tasks:Running task internal.archive... 2021-05-12T11:42:23.707 INFO:teuthology.task.internal:Creating archive directory... 2021-05-12T11:42:23.708 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-05-12T11:42:23.731 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-05-12T11:42:23.741 INFO:teuthology.task.internal:Enabling coredump saving... 2021-05-12T11:42:23.741 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-05-12T11:42:23.798 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:42:23.808 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:42:23.809 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-05-12T11:42:23.818 INFO:teuthology.task.internal:Configuring sudo... 2021-05-12T11:42:23.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-05-12T11:42:23.874 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-05-12T11:42:23.889 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-05-12T11:42:23.890 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-05-12T11:42:23.928 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-12T11:42:24.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-12T11:42:24.059 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-12T11:42:24.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-12T11:42:24.179 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2021-05-12T11:42:24.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-05-12T11:42:24.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2021-05-12T11:42:24.351 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-05-12T11:42:24.805 INFO:teuthology.run_tasks:Running task internal.timer... 2021-05-12T11:42:24.815 INFO:teuthology.task.internal:Starting timer... 2021-05-12T11:42:24.815 INFO:teuthology.run_tasks:Running task pcp... 2021-05-12T11:42:24.852 INFO:teuthology.run_tasks:Running task selinux... 2021-05-12T11:42:24.881 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-05-12T11:42:24.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2021-05-12T11:42:24.970 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: [ OK ] 2021-05-12T11:42:24.972 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-05-12T11:42:24.973 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2021-05-12T11:42:25.004 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2021-05-12T11:42:25.004 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2021-05-12T11:42:25.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-05-12T11:42:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-05-12T11:42:25.070 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-05-12T11:42:25.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2021-05-12T11:42:25.147 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-05-12T11:42:25.158 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-05-12T11:42:25.159 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-05-12T11:42:25.178 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-05-12T11:42:25.180 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-05-12T11:42:25.532 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-05-12T11:42:26.338 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-05-12T11:42:28.403 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-05-12T11:42:28.424 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:28 +0000 (0:00:00.037) 0:00:00.037 ********* 2021-05-12T11:42:29.816 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi191.front.sepia.ceph.com] 2021-05-12T11:42:29.871 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:29 +0000 (0:00:01.447) 0:00:01.484 ********* 2021-05-12T11:42:30.177 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-05-12T11:42:30.178 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-05-12T11:42:30.198 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:30 +0000 (0:00:00.326) 0:00:01.810 ********* 2021-05-12T11:42:30.773 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-05-12T11:42:30.775 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-05-12T11:42:30.796 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:30 +0000 (0:00:00.598) 0:00:02.409 ********* 2021-05-12T11:42:30.854 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-05-12T11:42:30.856 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:42:30.875 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:30 +0000 (0:00:00.079) 0:00:02.488 ********* 2021-05-12T11:42:31.182 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi191.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:42:31.204 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:31 +0000 (0:00:00.328) 0:00:02.816 ********* 2021-05-12T11:42:32.165 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-05-12T11:42:32.167 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620819751.2488806-12219-160733179221372/source", "state": "file", "uid": 0} 2021-05-12T11:42:32.186 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:32 +0000 (0:00:00.982) 0:00:03.799 ********* 2021-05-12T11:42:32.845 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-05-12T11:42:32.847 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-05-12T11:42:32.849 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-05-12T11:42:32.911 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-05-12T11:42:32.914 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-05-12T11:42:33.103 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:33 +0000 (0:00:00.916) 0:00:04.716 ********* 2021-05-12T11:42:33.707 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi191.front.sepia.ceph.com] 2021-05-12T11:42:33.756 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Wednesday 12 May 2021 11:42:33 +0000 (0:00:00.652) 0:00:05.369 ********* 2021-05-12T11:42:34.392 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-05-12T11:42:34.393 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.174802", "end": "2021-05-12 11:42:34.368712", "rc": 0, "start": "2021-05-12 11:42:34.193910", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-05-12T11:42:34.413 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:34 +0000 (0:00:00.657) 0:00:06.026 ********* 2021-05-12T11:42:34.605 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-05-12T11:42:34.607 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014150", "end": "2021-05-12 11:42:34.587597", "failed_when_result": false, "rc": 0, "start": "2021-05-12 11:42:34.573447", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-05-12T11:42:34.627 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:34 +0000 (0:00:00.213) 0:00:06.240 ********* 2021-05-12T11:42:34.694 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-05-12T11:42:34.696 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:42:34.716 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:34 +0000 (0:00:00.089) 0:00:06.329 ********* 2021-05-12T11:42:34.783 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-05-12T11:42:34.785 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:42:34.807 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:34 +0000 (0:00:00.090) 0:00:06.419 ********* 2021-05-12T11:42:34.874 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-05-12T11:42:34.876 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:42:34.895 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:34 +0000 (0:00:00.088) 0:00:06.508 ********* 2021-05-12T11:42:34.962 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-05-12T11:42:34.964 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:42:34.984 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:34 +0000 (0:00:00.088) 0:00:06.596 ********* 2021-05-12T11:42:35.050 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-05-12T11:42:35.052 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:42:35.071 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:35 +0000 (0:00:00.087) 0:00:06.683 ********* 2021-05-12T11:42:35.293 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi191.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:42:35.316 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:42:35 +0000 (0:00:00.244) 0:00:06.928 ********* 2021-05-12T11:42:35.602 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-05-12T11:42:35.604 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-05-12T11:42:35.801 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-05-12T11:42:35.988 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-05-12T11:42:36.179 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-05-12T11:42:36.376 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-05-12T11:42:36.562 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-05-12T11:42:36.749 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-05-12T11:42:36.945 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-05-12T11:42:37.131 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-05-12T11:42:37.331 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-05-12T11:42:37.517 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-05-12T11:42:37.717 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-05-12T11:42:37.915 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-05-12T11:42:38.102 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-05-12T11:42:38.289 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-05-12T11:42:38.486 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-05-12T11:42:38.673 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-05-12T11:42:38.869 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-05-12T11:42:39.055 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-05-12T11:42:39.242 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-05-12T11:42:39.440 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-05-12T11:42:39.626 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-05-12T11:42:39.823 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-05-12T11:42:40.009 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-05-12T11:42:40.196 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-05-12T11:42:40.392 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-05-12T11:42:40.578 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-05-12T11:42:40.775 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-05-12T11:42:40.965 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-05-12T11:42:41.206 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-05-12T11:42:41.399 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-05-12T11:42:41.600 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-05-12T11:42:41.790 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-05-12T11:42:41.989 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-05-12T11:42:42.175 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-05-12T11:42:42.373 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-05-12T11:42:42.559 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-05-12T11:42:42.757 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-05-12T11:42:42.944 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-05-12T11:42:43.131 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-05-12T11:42:43.328 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-05-12T11:42:43.514 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-05-12T11:42:43.712 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-05-12T11:42:43.909 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-05-12T11:42:44.096 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-05-12T11:42:44.283 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-05-12T11:42:44.480 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-05-12T11:42:44.668 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-05-12T11:42:44.866 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-05-12T11:42:45.053 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-05-12T11:42:45.241 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-05-12T11:42:45.437 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-05-12T11:42:45.624 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-05-12T11:42:45.821 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-05-12T11:42:46.017 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-05-12T11:42:46.214 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-05-12T11:42:46.400 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-05-12T11:42:46.586 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-05-12T11:42:46.782 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-05-12T11:42:46.980 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-05-12T11:42:47.166 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-05-12T11:42:47.352 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-05-12T11:42:47.549 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-05-12T11:42:47.735 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-05-12T11:42:47.933 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-05-12T11:42:48.120 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-05-12T11:42:48.316 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-05-12T11:42:48.503 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-05-12T11:42:48.701 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-05-12T11:42:48.888 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-05-12T11:42:49.085 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-05-12T11:42:49.283 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-05-12T11:42:49.469 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-05-12T11:42:49.657 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-05-12T11:42:49.848 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-05-12T11:42:50.046 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-05-12T11:42:50.233 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-05-12T11:42:50.422 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-05-12T11:42:50.609 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-05-12T11:42:50.807 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-05-12T11:42:50.996 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-05-12T11:42:51.185 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-05-12T11:42:51.372 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-05-12T11:42:51.568 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-05-12T11:42:51.764 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-05-12T11:42:51.952 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-05-12T11:42:52.148 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-05-12T11:42:52.348 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-05-12T11:42:52.535 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-05-12T11:42:52.732 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-05-12T11:42:52.921 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-05-12T11:42:53.117 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-05-12T11:42:53.304 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-05-12T11:42:53.500 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-05-12T11:42:53.687 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-05-12T11:42:53.884 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-05-12T11:42:54.072 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-05-12T11:42:54.268 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-05-12T11:42:54.455 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-05-12T11:42:54.652 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-05-12T11:42:54.839 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-05-12T11:42:55.037 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-05-12T11:42:55.223 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-05-12T11:42:55.420 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-05-12T11:42:55.605 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-05-12T11:42:55.802 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-05-12T11:42:55.988 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-05-12T11:42:56.184 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-05-12T11:42:56.370 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-05-12T11:42:56.566 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-05-12T11:42:56.752 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-05-12T11:42:56.949 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-05-12T11:42:57.136 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-05-12T11:42:57.322 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-05-12T11:42:57.518 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-05-12T11:42:57.705 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-05-12T11:42:57.902 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-05-12T11:42:58.091 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-05-12T11:42:58.289 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-05-12T11:42:58.479 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-05-12T11:42:58.668 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-05-12T11:42:58.866 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-05-12T11:42:59.053 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-05-12T11:42:59.240 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-05-12T11:42:59.437 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-05-12T11:42:59.634 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-05-12T11:42:59.820 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-05-12T11:43:00.013 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-05-12T11:43:00.202 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-05-12T11:43:00.389 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-05-12T11:43:00.587 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-05-12T11:43:00.774 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-05-12T11:43:00.960 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-05-12T11:43:01.157 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-05-12T11:43:02.664 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-05-12T11:43:04.223 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-05-12T11:43:05.733 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-05-12T11:43:07.198 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-05-12T11:43:08.659 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-05-12T11:43:08.918 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:08 +0000 (0:00:33.601) 0:00:40.530 ********* 2021-05-12T11:43:08.985 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-05-12T11:43:09.006 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:09 +0000 (0:00:00.089) 0:00:40.619 ********* 2021-05-12T11:43:09.212 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-05-12T11:43:09.213 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-05-12T11:43:09.215 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-05-12T11:43:09.217 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-05-12T11:43:09.218 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-05-12T11:43:09.220 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de 2021-05-12T11:43:09.222 INFO:teuthology.task.ansible.out:53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d 2021-05-12T11:43:09.223 INFO:teuthology.task.ansible.out:7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846ab 2021-05-12T11:43:09.224 INFO:teuthology.task.ansible.out:d9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumra 2021-05-12T11:43:09.225 INFO:teuthology.task.ansible.out:o", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b76 2021-05-12T11:43:09.227 INFO:teuthology.task.ansible.out:8c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70f 2021-05-12T11:43:09.228 INFO:teuthology.task.ansible.out:f00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41 2021-05-12T11:43:09.230 INFO:teuthology.task.ansible.out:e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5 2021-05-12T11:43:09.232 INFO:teuthology.task.ansible.out:dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b 2021-05-12T11:43:09.233 INFO:teuthology.task.ansible.out:1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpiv 2021-05-12T11:43:09.234 INFO:teuthology.task.ansible.out:onka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}]}, "changed": false} 2021-05-12T11:43:09.260 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:09 +0000 (0:00:00.253) 0:00:40.872 ********* 2021-05-12T11:43:10.114 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-05-12T11:43:10.116 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"after": "2f6366fd31821994f3d217ad18c1675bf42a2115", "attempts": 1, "before": "2f6366fd31821994f3d217ad18c1675bf42a2115", "changed": false, "remote_url_changed": false} 2021-05-12T11:43:10.135 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:10 +0000 (0:00:00.875) 0:00:41.748 ********* 2021-05-12T11:43:10.482 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-05-12T11:43:10.483 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-05-12T11:43:10.710 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-05-12T11:43:10.960 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-05-12T11:43:11.188 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-05-12T11:43:11.190 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-05-12T11:43:11.439 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-05-12T11:43:11.680 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-05-12T11:43:11.682 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-05-12T11:43:11.920 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-05-12T11:43:12.249 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-05-12T11:43:12.510 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-05-12T11:43:12.636 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:12.863 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-05-12T11:43:12.865 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-05-12T11:43:13.090 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-05-12T11:43:13.092 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:13.102 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:13.329 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-05-12T11:43:13.567 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-05-12T11:43:13.569 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-05-12T11:43:13.794 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-05-12T11:43:14.014 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-05-12T11:43:14.016 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-05-12T11:43:14.242 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-05-12T11:43:14.469 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-05-12T11:43:14.697 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-05-12T11:43:14.925 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-05-12T11:43:14.927 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-05-12T11:43:15.162 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-05-12T11:43:15.164 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-05-12T11:43:15.410 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-05-12T11:43:15.637 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-05-12T11:43:15.864 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-05-12T11:43:15.866 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-05-12T11:43:16.092 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-05-12T11:43:16.320 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-05-12T11:43:16.547 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-05-12T11:43:16.775 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-05-12T11:43:17.002 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-05-12T11:43:17.231 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-05-12T11:43:17.273 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-05-12T11:43:17.458 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-05-12T11:43:17.686 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-05-12T11:43:17.914 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-05-12T11:43:18.143 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-05-12T11:43:18.371 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-05-12T11:43:18.599 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-12T11:43:18.818 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-05-12T11:43:19.065 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-05-12T11:43:19.292 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-05-12T11:43:19.294 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-05-12T11:43:19.531 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-05-12T11:43:19.534 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-05-12T11:43:19.758 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-05-12T11:43:19.986 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-05-12T11:43:20.217 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-05-12T11:43:20.465 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtV 2021-05-12T11:43:20.468 INFO:teuthology.task.ansible.out:TFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-05-12T11:43:20.704 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-05-12T11:43:20.706 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-05-12T11:43:20.708 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-05-12T11:43:20.921 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-05-12T11:43:21.149 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-05-12T11:43:21.379 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-05-12T11:43:21.606 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-05-12T11:43:21.836 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-05-12T11:43:22.068 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-05-12T11:43:22.070 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-05-12T11:43:22.287 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-05-12T11:43:22.517 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-05-12T11:43:22.738 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-05-12T11:43:22.740 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-05-12T11:43:22.969 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-05-12T11:43:23.210 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-05-12T11:43:23.438 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-05-12T11:43:23.440 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-05-12T11:43:23.666 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-05-12T11:43:23.904 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-05-12T11:43:24.135 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-05-12T11:43:24.353 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-12T11:43:24.581 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-05-12T11:43:24.808 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-05-12T11:43:25.036 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-05-12T11:43:25.264 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-05-12T11:43:25.492 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-05-12T11:43:25.720 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-05-12T11:43:25.948 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-05-12T11:43:26.168 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-05-12T11:43:26.418 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-05-12T11:43:26.658 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-05-12T11:43:26.886 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-05-12T11:43:27.113 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-05-12T11:43:27.342 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-05-12T11:43:27.581 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-05-12T11:43:27.808 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-05-12T11:43:28.037 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-05-12T11:43:28.275 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-05-12T11:43:28.276 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-05-12T11:43:28.502 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-05-12T11:43:28.742 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-05-12T11:43:28.970 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-05-12T11:43:29.187 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-05-12T11:43:29.416 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-05-12T11:43:29.417 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-05-12T11:43:29.642 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-05-12T11:43:29.644 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-05-12T11:43:29.870 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-05-12T11:43:30.110 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-05-12T11:43:30.338 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-05-12T11:43:30.566 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-05-12T11:43:30.793 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-05-12T11:43:31.013 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-05-12T11:43:31.251 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-05-12T11:43:31.478 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-05-12T11:43:31.480 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-05-12T11:43:31.705 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-05-12T11:43:31.933 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-05-12T11:43:32.164 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-05-12T11:43:32.391 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-05-12T11:43:32.610 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-05-12T11:43:32.838 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-05-12T11:43:33.072 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-05-12T11:43:33.074 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-05-12T11:43:33.301 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-05-12T11:43:33.529 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-05-12T11:43:33.758 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-05-12T11:43:33.976 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-05-12T11:43:34.206 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-05-12T11:43:34.444 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-05-12T11:43:34.663 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-05-12T11:43:34.890 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-05-12T11:43:35.117 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-05-12T11:43:35.345 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-05-12T11:43:35.583 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-05-12T11:43:35.813 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-05-12T11:43:36.030 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-05-12T11:43:36.261 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-05-12T11:43:36.493 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-05-12T11:43:36.732 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-05-12T11:43:36.961 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-05-12T11:43:37.189 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-05-12T11:43:37.419 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-05-12T11:43:37.420 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-05-12T11:43:37.646 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-05-12T11:43:37.887 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-05-12T11:43:38.120 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-05-12T11:43:38.350 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-05-12T11:43:38.568 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-05-12T11:43:38.796 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-05-12T11:43:39.024 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-05-12T11:43:39.262 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-05-12T11:43:39.480 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-05-12T11:43:39.707 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-05-12T11:43:39.935 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-05-12T11:43:40.163 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-05-12T11:43:40.390 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-05-12T11:43:40.628 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-05-12T11:43:40.856 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-05-12T11:43:41.083 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-05-12T11:43:41.321 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-05-12T11:43:41.322 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-05-12T11:43:41.549 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-05-12T11:43:41.776 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-05-12T11:43:41.949 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:41 +0000 (0:00:31.813) 0:01:13.561 ********* 2021-05-12T11:43:42.028 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.040 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.040 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.052 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.053 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.063 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.064 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.075 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.085 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.317 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-05-12T11:43:42.328 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.329 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.566 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-05-12T11:43:42.795 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-05-12T11:43:42.806 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.910 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ub 2021-05-12T11:43:42.912 INFO:teuthology.task.ansible.out:untu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a 2021-05-12T11:43:42.915 INFO:teuthology.task.ansible.out:3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizh 2021-05-12T11:43:42.918 INFO:teuthology.task.ansible.out:an@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) = 2021-05-12T11:43:42.920 INFO:teuthology.task.ansible.out:> {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name 2021-05-12T11:43:42.921 INFO:teuthology.task.ansible.out:": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f 2021-05-12T11:43:42.923 INFO:teuthology.task.ansible.out:0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.928 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.939 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.950 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.961 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.963 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.974 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.986 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:42.998 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.009 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.022 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.033 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.034 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.044 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.055 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.058 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.067 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.069 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.079 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.081 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.091 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.092 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.103 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.103 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.115 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.115 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.127 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.128 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.139 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.151 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.153 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.164 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.165 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.176 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.188 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.189 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.199 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.201 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.211 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.211 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.222 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.224 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.233 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.244 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.246 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.255 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.257 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.267 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.271 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.278 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.289 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.291 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.300 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.303 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.312 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.314 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.323 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.334 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.336 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.345 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.347 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.357 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.367 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.368 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.368 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.380 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.391 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.392 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.402 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.404 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.413 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.415 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.425 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.436 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.438 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.451 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.453 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.460 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.461 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.472 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.473 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.484 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.485 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.497 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.498 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.512 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.514 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.528 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.530 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.542 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.544 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.554 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.565 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.567 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.577 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.587 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.590 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.599 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.610 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.648 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:43 +0000 (0:00:01.698) 0:01:15.260 ********* 2021-05-12T11:43:43.715 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:43.736 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:43 +0000 (0:00:00.088) 0:01:15.348 ********* 2021-05-12T11:43:44.021 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-05-12T11:43:44.022 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-05-12T11:43:44.210 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-05-12T11:43:44.406 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-05-12T11:43:44.592 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-05-12T11:43:44.778 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-05-12T11:43:44.965 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-05-12T11:43:45.161 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-05-12T11:43:45.347 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-05-12T11:43:45.533 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-05-12T11:43:45.719 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-05-12T11:43:45.906 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-05-12T11:43:46.094 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-05-12T11:43:46.281 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-05-12T11:43:46.468 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-05-12T11:43:46.666 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-05-12T11:43:46.853 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-05-12T11:43:47.040 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-05-12T11:43:47.226 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-05-12T11:43:47.412 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-05-12T11:43:47.599 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-05-12T11:43:47.787 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-05-12T11:43:47.976 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-05-12T11:43:48.163 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-05-12T11:43:48.359 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-05-12T11:43:48.545 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-05-12T11:43:48.732 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-05-12T11:43:48.930 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-05-12T11:43:49.117 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-05-12T11:43:49.313 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-05-12T11:43:49.500 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-05-12T11:43:49.687 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-05-12T11:43:49.874 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-05-12T11:43:50.061 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-05-12T11:43:50.248 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-05-12T11:43:50.445 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-05-12T11:43:50.642 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-05-12T11:43:52.243 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-05-12T11:43:53.573 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-05-12T11:43:54.922 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-05-12T11:43:55.128 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-05-12T11:43:55.324 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-05-12T11:43:55.511 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-05-12T11:43:55.696 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-05-12T11:43:55.882 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-05-12T11:43:56.078 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-05-12T11:43:56.265 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-05-12T11:43:56.455 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-05-12T11:43:56.640 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-05-12T11:43:56.837 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-05-12T11:43:56.877 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:56 +0000 (0:00:13.141) 0:01:28.490 ********* 2021-05-12T11:43:57.409 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-05-12T11:43:57.411 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "checksum": "313dfb187fdfe6062591abb65b175f610cf74a8a", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "b87ac93b406a130b21ee0476c1efad05", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620819836.9638984-18365-216299281345654/source", "state": "file", "uid": 0} 2021-05-12T11:43:57.431 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:57 +0000 (0:00:00.553) 0:01:29.043 ********* 2021-05-12T11:43:57.508 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-05-12T11:43:57.509 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => { "msg": "Host smithi191.front.sepia.ceph.com is running RedHat 8.3 (Ootpa)" } 2021-05-12T11:43:57.530 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:57 +0000 (0:00:00.099) 0:01:29.142 ********* 2021-05-12T11:43:57.598 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-05-12T11:43:57.600 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-05-12T11:43:57.620 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:57 +0000 (0:00:00.090) 0:01:29.233 ********* 2021-05-12T11:43:57.687 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-05-12T11:43:57.688 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:57.708 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:57 +0000 (0:00:00.087) 0:01:29.321 ********* 2021-05-12T11:43:58.281 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-05-12T11:43:58.282 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.342054", "end": "2021-05-12 11:43:58.251043", "rc": 0, "start": "2021-05-12 11:43:57.908989", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-05-12T11:43:58.303 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:58 +0000 (0:00:00.595) 0:01:29.916 ********* 2021-05-12T11:43:58.383 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:58.405 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:58 +0000 (0:00:00.101) 0:01:30.018 ********* 2021-05-12T11:43:58.473 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-05-12T11:43:58.475 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:58.494 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:58 +0000 (0:00:00.088) 0:01:30.107 ********* 2021-05-12T11:43:58.561 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-05-12T11:43:58.563 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:58.583 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:58 +0000 (0:00:00.088) 0:01:30.195 ********* 2021-05-12T11:43:58.649 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-05-12T11:43:58.651 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:58.672 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:58 +0000 (0:00:00.089) 0:01:30.285 ********* 2021-05-12T11:43:58.740 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-05-12T11:43:58.742 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:43:58.761 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:58 +0000 (0:00:00.089) 0:01:30.374 ********* 2021-05-12T11:43:58.838 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-05-12T11:43:58.841 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-05-12T11:43:58.861 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:58 +0000 (0:00:00.099) 0:01:30.474 ********* 2021-05-12T11:43:58.949 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-05-12T11:43:58.950 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi191.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T11:43:58.973 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:58 +0000 (0:00:00.112) 0:01:30.586 ********* 2021-05-12T11:43:59.050 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-05-12T11:43:59.052 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-05-12T11:43:59.073 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:43:59 +0000 (0:00:00.099) 0:01:30.685 ********* 2021-05-12T11:44:00.274 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-05-12T11:44:00.276 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1620819840.1425884-tcqadzr2/katello-ca-consumer-latest.noarchud4137ko.rpm"]} 2021-05-12T11:44:00.296 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:00 +0000 (0:00:01.222) 0:01:31.908 ********* 2021-05-12T11:44:00.826 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "checksum": "a626a25b7b500f9214acf310b1376506a11f0de8", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "md5sum": "5e20d6ed7744f334ad7deb3cea4d5905", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 3, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620819840.4016201-18496-51211787110441/source", "state": "file", "uid": 0} 2021-05-12T11:44:00.847 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:00 +0000 (0:00:00.551) 0:01:32.460 ********* 2021-05-12T11:44:01.429 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-05-12T11:44:01.431 INFO:teuthology.task.ansible.out:fatal: [smithi191.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-05-12T11:44:01.451 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:01 +0000 (0:00:00.603) 0:01:33.064 ********* 2021-05-12T11:44:01.518 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-05-12T11:44:01.520 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-05-12T11:44:01.540 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:01 +0000 (0:00:00.089) 0:01:33.153 ********* 2021-05-12T11:44:17.301 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-05-12T11:44:17.303 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-05-12T11:44:17.323 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:17 +0000 (0:00:15.782) 0:01:48.935 ********* 2021-05-12T11:44:17.400 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-05-12T11:44:17.402 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-05-12T11:44:17.422 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:17 +0000 (0:00:00.099) 0:01:49.035 ********* 2021-05-12T11:44:18.111 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-05-12T11:44:18.113 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.448910", "end": "2021-05-12 11:44:18.079629", "failed_when_result": false, "rc": 0, "start": "2021-05-12 11:44:17.630719", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-05-12T11:44:18.137 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:18 +0000 (0:00:00.715) 0:01:49.750 ********* 2021-05-12T11:44:18.207 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-05-12T11:44:18.208 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:18.229 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:18 +0000 (0:00:00.091) 0:01:49.841 ********* 2021-05-12T11:44:18.296 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-05-12T11:44:18.298 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:18.318 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:18 +0000 (0:00:00.089) 0:01:49.931 ********* 2021-05-12T11:44:18.375 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi191.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T11:44:18.397 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:18 +0000 (0:00:00.078) 0:01:50.010 ********* 2021-05-12T11:44:18.465 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:18.489 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:18 +0000 (0:00:00.092) 0:01:50.102 ********* 2021-05-12T11:44:21.089 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-05-12T11:44:21.091 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.372810", "end": "2021-05-12 11:44:21.063548", "rc": 0, "start": "2021-05-12 11:44:18.690738", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-05-12T11:44:21.112 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:21 +0000 (0:00:02.622) 0:01:52.724 ********* 2021-05-12T11:44:21.190 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-05-12T11:44:21.192 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-05-12T11:44:21.214 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:21 +0000 (0:00:00.101) 0:01:52.826 ********* 2021-05-12T11:44:21.282 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-05-12T11:44:21.283 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:21.303 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:21 +0000 (0:00:00.089) 0:01:52.916 ********* 2021-05-12T11:44:21.382 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-05-12T11:44:21.384 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-05-12T11:44:21.403 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:21 +0000 (0:00:00.099) 0:01:53.016 ********* 2021-05-12T11:44:21.470 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-05-12T11:44:21.472 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-05-12T11:44:21.502 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:21 +0000 (0:00:00.098) 0:01:53.115 ********* 2021-05-12T11:44:23.601 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-05-12T11:44:23.602 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-05-12T11:44:23.623 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:23 +0000 (0:00:02.120) 0:01:55.235 ********* 2021-05-12T11:44:26.972 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.117517", "end": "2021-05-12 11:44:26.947763", "rc": 0, "start": "2021-05-12 11:44:23.830246", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-05-12T11:44:26.992 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:26 +0000 (0:00:03.369) 0:01:58.605 ********* 2021-05-12T11:44:27.485 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-05-12T11:44:27.487 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-05-12T11:44:27.506 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:27 +0000 (0:00:00.514) 0:01:59.119 ********* 2021-05-12T11:44:27.739 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi191.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:44:27.760 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:27 +0000 (0:00:00.254) 0:01:59.373 ********* 2021-05-12T11:44:28.255 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-05-12T11:44:28.257 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-05-12T11:44:28.679 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-05-12T11:44:28.702 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:28 +0000 (0:00:00.941) 0:02:00.315 ********* 2021-05-12T11:44:29.224 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-05-12T11:44:29.226 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-05-12T11:44:29.660 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-05-12T11:44:29.684 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:29 +0000 (0:00:00.981) 0:02:01.296 ********* 2021-05-12T11:44:29.751 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-05-12T11:44:29.753 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:29.774 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:29 +0000 (0:00:00.089) 0:02:01.386 ********* 2021-05-12T11:44:29.841 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-05-12T11:44:29.843 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:29.863 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:29 +0000 (0:00:00.089) 0:02:01.476 ********* 2021-05-12T11:44:29.931 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-05-12T11:44:29.933 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:29.953 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:29 +0000 (0:00:00.089) 0:02:01.566 ********* 2021-05-12T11:44:30.021 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-05-12T11:44:30.023 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:30.043 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:30 +0000 (0:00:00.089) 0:02:01.655 ********* 2021-05-12T11:44:30.111 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-05-12T11:44:30.113 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-05-12T11:44:30.122 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-05-12T11:44:30.124 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-05-12T11:44:30.145 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:30 +0000 (0:00:00.102) 0:02:01.758 ********* 2021-05-12T11:44:30.213 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-05-12T11:44:30.215 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:30.237 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:30 +0000 (0:00:00.091) 0:02:01.850 ********* 2021-05-12T11:44:30.294 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-05-12T11:44:30.296 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:30.316 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:30 +0000 (0:00:00.078) 0:02:01.929 ********* 2021-05-12T11:44:30.373 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-05-12T11:44:30.375 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:30.394 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:30 +0000 (0:00:00.078) 0:02:02.007 ********* 2021-05-12T11:44:30.462 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-05-12T11:44:30.463 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:30.483 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:30 +0000 (0:00:00.088) 0:02:02.096 ********* 2021-05-12T11:44:30.541 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-05-12T11:44:30.543 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:30.564 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:30 +0000 (0:00:00.080) 0:02:02.176 ********* 2021-05-12T11:44:31.520 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-05-12T11:44:31.521 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.766068", "end": "2021-05-12 11:44:31.500342", "rc": 0, "start": "2021-05-12 11:44:30.734274", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:44:31.542 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:31 +0000 (0:00:00.978) 0:02:03.155 ********* 2021-05-12T11:44:43.083 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-05-12T11:44:43.085 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:44:43.105 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:43 +0000 (0:00:11.562) 0:02:14.717 ********* 2021-05-12T11:44:43.172 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-05-12T11:44:43.173 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:43.193 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:43 +0000 (0:00:00.088) 0:02:14.805 ********* 2021-05-12T11:44:43.260 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-05-12T11:44:43.262 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:43.281 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:43 +0000 (0:00:00.088) 0:02:14.894 ********* 2021-05-12T11:44:43.348 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:43.369 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:43 +0000 (0:00:00.088) 0:02:14.982 ********* 2021-05-12T11:44:43.436 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-05-12T11:44:43.438 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:43.458 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:43 +0000 (0:00:00.088) 0:02:15.070 ********* 2021-05-12T11:44:43.939 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-05-12T11:44:43.941 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-05-12T11:44:43.961 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:43 +0000 (0:00:00.502) 0:02:15.573 ********* 2021-05-12T11:44:46.523 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-05-12T11:44:46.525 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:44:46.544 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:46 +0000 (0:00:02.583) 0:02:18.157 ********* 2021-05-12T11:44:47.051 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-05-12T11:44:47.053 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-05-12T11:44:47.478 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-05-12T11:44:47.893 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-05-12T11:44:47.917 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:47 +0000 (0:00:01.372) 0:02:19.530 ********* 2021-05-12T11:44:48.098 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-05-12T11:44:48.099 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T11:44:48.120 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:48 +0000 (0:00:00.202) 0:02:19.732 ********* 2021-05-12T11:44:48.664 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-05-12T11:44:48.666 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620819888.2101014-21455-275480054730854/source", "state": "file", "uid": 0} 2021-05-12T11:44:49.068 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-05-12T11:44:49.460 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-05-12T11:44:49.483 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:49 +0000 (0:00:01.363) 0:02:21.096 ********* 2021-05-12T11:44:49.551 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-05-12T11:44:49.552 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:44:49.572 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:49 +0000 (0:00:00.088) 0:02:21.185 ********* 2021-05-12T11:44:49.649 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-05-12T11:44:49.651 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-05-12T11:44:49.671 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:49 +0000 (0:00:00.099) 0:02:21.284 ********* 2021-05-12T11:44:49.894 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-05-12T11:44:49.935 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} Wednesday 12 May 2021 11:44:49 +0000 (0:00:00.243) 0:02:21.528 ********* 2021-05-12T11:44:49.994 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-05-12T11:44:49.996 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T11:44:50.005 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T11:44:50.016 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T11:44:50.038 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:44:50 +0000 (0:00:00.123) 0:02:21.651 ********* 2021-05-12T11:45:07.340 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-05-12T11:45:07.342 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-plugins-load-2.3.3-5.el8.x86_64", "Installed: nagios-common-4.4.6-4.el8.x86_64", "Installed: nrpe-4.0.3-6.el8.x86_64", "Installed: nagios-plugins-2.3.3-5.el8.x86_64", "Removed: nrpe-4.0.3-1.el8.x86_64", "Removed: nagios-common-4.4.5-1.el8.x86_64", "Removed: nagios-plugins-2.3.3-4.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-4.el8.x86_64"]} 2021-05-12T11:45:07.361 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:07 +0000 (0:00:17.322) 0:02:38.974 ********* 2021-05-12T11:45:07.428 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-05-12T11:45:07.430 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:45:07.458 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:07 +0000 (0:00:00.096) 0:02:39.071 ********* 2021-05-12T11:45:07.941 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-05-12T11:45:07.943 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-05-12T11:45:07.963 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:07 +0000 (0:00:00.505) 0:02:39.576 ********* 2021-05-12T11:45:08.453 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-05-12T11:45:08.455 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-05-12T11:45:08.475 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:08 +0000 (0:00:00.511) 0:02:40.088 ********* 2021-05-12T11:45:08.543 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-05-12T11:45:08.545 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-05-12T11:45:08.554 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-05-12T11:45:08.576 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:08 +0000 (0:00:00.101) 0:02:40.189 ********* 2021-05-12T11:45:08.808 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-05-12T11:45:08.809 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:45:08.830 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:08 +0000 (0:00:00.253) 0:02:40.442 ********* 2021-05-12T11:45:09.123 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-05-12T11:45:09.125 INFO:teuthology.task.ansible.out:fatal: [smithi191.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-05-12T11:45:09.144 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:09 +0000 (0:00:00.314) 0:02:40.757 ********* 2021-05-12T11:45:09.212 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-05-12T11:45:09.213 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:45:09.233 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:09 +0000 (0:00:00.088) 0:02:40.846 ********* 2021-05-12T11:45:09.713 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-05-12T11:45:09.714 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-05-12T11:45:09.736 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:09 +0000 (0:00:00.502) 0:02:41.349 ********* 2021-05-12T11:45:10.467 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-05-12T11:45:10.469 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-05-12 11:44:55 UTC", "ActiveEnterTimestampMonotonic": "215838916", "ActiveExitTimestamp": "Wed 2021-05-12 11:44:55 UTC", "ActiveExitTimestampMonotonic": "215811771", "ActiveState": "active", "After": "system.slice nss-lookup.target local-fs.target tmp.mount sysinit.target basic.target time-sync.target systemd-journald.socket systemd-tmpfiles-setup.service -.mount var-run.mount network-online.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-05-12 11:44:55 UTC", "AssertTimestampMonotonic": "215838137", "Before": "getty@tty1.service plymouth-quit.service shutdown.target xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-05-12 11:44:55 UTC", "Co 2021-05-12T11:45:10.470 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "215838137", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "16064", "ExecMainStartTimestamp": "Wed 2021-05-12 11:44:55 UTC", "ExecMainStartTimestampMonotonic": "215838892", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "thawing", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Wed 2021-05-12 11:44:55 UTC", "InactiveEnterTimestampMonotonic": "215837379", "InactiveExitTimestamp": "Wed 2021-05-12 11:44:55 UTC", 2021-05-12T11:45:10.472 INFO:teuthology.task.ansible.out:"InactiveExitTimestampMonotonic": "215838916", "InvocationID": "df488059afd04a1a858769505749b698", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127238", "LimitNPROCSoft": "127238", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127238", "LimitSIGPENDINGSoft": "127238", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "16064", "MemoryAccounting": "yes", "MemoryCurrent": "888832", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateT 2021-05-12T11:45:10.474 INFO:teuthology.task.ansible.out:mp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-05-12 11:44:55 UTC", "StateChangeTimestampMonotonic": "215838916", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-05-12 11:44:55 UTC", "WatchdogTimestampMonotonic": "215838915", "WatchdogUSec": "0"}} 2021-05-12T11:45:10.490 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:10 +0000 (0:00:00.753) 0:02:42.102 ********* 2021-05-12T11:45:10.711 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-05-12T11:45:10.713 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001684", "end": "2021-05-12 11:45:10.685935", "rc": 0, "start": "2021-05-12 11:45:10.684251", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-05-12T11:45:10.732 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:10 +0000 (0:00:00.242) 0:02:42.345 ********* 2021-05-12T11:45:21.934 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-05-12T11:45:21.937 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:45:21.957 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:45:21 +0000 (0:00:11.224) 0:02:53.570 ********* 2021-05-12T11:46:11.590 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-05-12T11:46:11.592 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-54.el8_3.4.noarch", "Installed: selinux-policy-3.14.3-54.el8_3.4.noarch", "Removed: selinux-policy-3.14.3-54.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8.noarch"]} 2021-05-12T11:46:11.612 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:11 +0000 (0:00:49.655) 0:03:43.225 ********* 2021-05-12T11:46:12.359 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-05-12T11:46:12.361 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-05-12T11:46:12.382 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:12 +0000 (0:00:00.769) 0:03:43.994 ********* 2021-05-12T11:46:28.838 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-05-12T11:46:28.840 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.235297", "end": "2021-05-12 11:46:28.814102", "failed_when_result": false, "rc": 0, "start": "2021-05-12 11:46:12.578805", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-05-12T11:46:28.860 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:28 +0000 (0:00:16.478) 0:04:00.473 ********* 2021-05-12T11:46:29.341 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-05-12T11:46:29.343 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-05-12T11:46:29.363 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:29 +0000 (0:00:00.503) 0:04:00.976 ********* 2021-05-12T11:46:29.596 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-05-12T11:46:29.598 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.009342", "end": "2021-05-12 11:46:29.573165", "rc": 0, "start": "2021-05-12 11:46:29.563823", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:46:29.618 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:29 +0000 (0:00:00.254) 0:04:01.230 ********* 2021-05-12T11:46:29.851 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-05-12T11:46:29.852 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.018237", "end": "2021-05-12 11:46:29.830252", "rc": 0, "start": "2021-05-12 11:46:29.812015", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:46:29.872 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:29 +0000 (0:00:00.254) 0:04:01.485 ********* 2021-05-12T11:46:46.316 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-05-12T11:46:46.318 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.215568", "end": "2021-05-12 11:46:46.286356", "rc": 0, "start": "2021-05-12 11:46:30.070788", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:46:46.338 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:46 +0000 (0:00:16.466) 0:04:17.951 ********* 2021-05-12T11:46:46.585 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-05-12T11:46:46.609 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:46 +0000 (0:00:00.270) 0:04:18.221 ********* 2021-05-12T11:46:46.677 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-05-12T11:46:46.701 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:46 +0000 (0:00:00.092) 0:04:18.313 ********* 2021-05-12T11:46:46.781 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-05-12T11:46:46.783 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-12T11:46:46.803 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:46 +0000 (0:00:00.102) 0:04:18.415 ********* 2021-05-12T11:46:46.893 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi191.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "podman-docker", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-05-12T11:46:46.914 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:46 +0000 (0:00:00.111) 0:04:18.527 ********* 2021-05-12T11:46:46.916 INFO:teuthology.task.ansible.out: 2021-05-12T11:46:46.983 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi191.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-12T11:46:47.005 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:47 +0000 (0:00:00.090) 0:04:18.618 ********* 2021-05-12T11:46:47.186 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-05-12T11:46:47.188 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-05-12T11:46:47.208 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:47 +0000 (0:00:00.202) 0:04:18.820 ********* 2021-05-12T11:46:47.429 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-05-12T11:46:47.431 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-05-12T11:46:47.452 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:47 +0000 (0:00:00.244) 0:04:19.065 ********* 2021-05-12T11:46:47.530 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-05-12T11:46:47.532 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:46:47.552 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:47 +0000 (0:00:00.099) 0:04:19.164 ********* 2021-05-12T11:46:47.619 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-05-12T11:46:47.621 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:46:47.641 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:47 +0000 (0:00:00.089) 0:04:19.253 ********* 2021-05-12T11:46:47.904 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-05-12T11:46:47.906 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-05-12T11:46:47.926 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:47 +0000 (0:00:00.284) 0:04:19.538 ********* 2021-05-12T11:46:48.483 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-05-12T11:46:48.485 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-05-12T11:46:48.505 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:48 +0000 (0:00:00.579) 0:04:20.118 ********* 2021-05-12T11:46:48.572 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-05-12T11:46:48.574 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi191"}, "changed": false} 2021-05-12T11:46:48.594 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:48 +0000 (0:00:00.088) 0:04:20.207 ********* 2021-05-12T11:46:49.625 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-05-12T11:46:49.627 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi191", "ansible_hostname": "smithi191", "ansible_nodename": "smithi191"}, "changed": false, "name": "smithi191"} 2021-05-12T11:46:49.648 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:49 +0000 (0:00:01.053) 0:04:21.260 ********* 2021-05-12T11:46:50.134 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-05-12T11:46:50.136 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-05-12T11:46:50.157 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:50 +0000 (0:00:00.509) 0:04:21.770 ********* 2021-05-12T11:46:50.628 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-05-12T11:46:50.630 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-05-12T11:46:50.650 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:50 +0000 (0:00:00.492) 0:04:22.262 ********* 2021-05-12T11:46:51.266 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi191.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-05-12T11:46:51.288 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:51 +0000 (0:00:00.638) 0:04:22.901 ********* 2021-05-12T11:46:51.777 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-05-12T11:46:51.779 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-05-12T11:46:51.800 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:51 +0000 (0:00:00.512) 0:04:23.413 ********* 2021-05-12T11:46:52.034 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-05-12T11:46:52.036 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-12T11:46:52.056 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:52 +0000 (0:00:00.255) 0:04:23.669 ********* 2021-05-12T11:46:52.279 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-05-12T11:46:52.281 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-05-12T11:46:52.300 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:52 +0000 (0:00:00.244) 0:04:23.913 ********* 2021-05-12T11:46:52.552 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-05-12T11:46:52.554 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-05-12T11:46:52.574 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:52 +0000 (0:00:00.273) 0:04:24.186 ********* 2021-05-12T11:46:53.065 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-05-12T11:46:53.067 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-05-12T11:46:53.089 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:53 +0000 (0:00:00.515) 0:04:24.702 ********* 2021-05-12T11:46:53.581 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-05-12T11:46:53.583 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-05-12T11:46:53.603 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:53 +0000 (0:00:00.514) 0:04:25.216 ********* 2021-05-12T11:46:55.975 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-05-12T11:46:55.976 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.146549", "end": "2021-05-12 11:46:55.945596", "rc": 0, "start": "2021-05-12 11:46:53.799047", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:46:55.997 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:55 +0000 (0:00:02.392) 0:04:27.609 ********* 2021-05-12T11:46:56.065 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-05-12T11:46:56.067 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:46:56.087 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:56 +0000 (0:00:00.090) 0:04:27.699 ********* 2021-05-12T11:46:56.610 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-05-12T11:46:56.612 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-05-12T11:46:56.635 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:56 +0000 (0:00:00.547) 0:04:28.247 ********* 2021-05-12T11:46:56.693 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-05-12T11:46:56.715 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:56 +0000 (0:00:00.079) 0:04:28.327 ********* 2021-05-12T11:46:56.783 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-05-12T11:46:56.805 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:56 +0000 (0:00:00.090) 0:04:28.418 ********* 2021-05-12T11:46:56.873 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:46:56.895 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:56 +0000 (0:00:00.089) 0:04:28.507 ********* 2021-05-12T11:46:56.963 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-05-12T11:46:56.964 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-05-12T11:46:56.974 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-05-12T11:46:56.996 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:56 +0000 (0:00:00.101) 0:04:28.609 ********* 2021-05-12T11:46:57.065 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-05-12T11:46:57.067 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:46:57.086 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:57 +0000 (0:00:00.089) 0:04:28.699 ********* 2021-05-12T11:46:57.155 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:46:57.177 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:57 +0000 (0:00:00.090) 0:04:28.790 ********* 2021-05-12T11:46:57.245 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:46:57.267 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:57 +0000 (0:00:00.089) 0:04:28.880 ********* 2021-05-12T11:46:57.337 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-05-12T11:46:57.338 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:46:57.360 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:57 +0000 (0:00:00.092) 0:04:28.973 ********* 2021-05-12T11:46:57.605 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-05-12T11:46:57.606 INFO:teuthology.task.ansible.out:fatal: [smithi191.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007145", "end": "2021-05-12 11:46:57.577932", "msg": "non-zero return code", "rc": 1, "start": "2021-05-12 11:46:57.570787", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-05-12T11:46:57.629 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:57 +0000 (0:00:00.269) 0:04:29.242 ********* 2021-05-12T11:46:57.698 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-05-12T11:46:57.700 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:46:57.722 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:46:57 +0000 (0:00:00.092) 0:04:29.335 ********* 2021-05-12T11:47:20.971 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-05-12T11:47:20.973 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2021-05-12T11:47:20.994 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:47:20 +0000 (0:00:23.271) 0:04:52.606 ********* 2021-05-12T11:47:44.040 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-05-12T11:47:44.042 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-05-12T11:47:44.063 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:47:44 +0000 (0:00:23.068) 0:05:15.675 ********* 2021-05-12T11:48:37.786 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-12T11:48:37.788 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: perl-Text-Unidecode-1.30-5.el8.noarch", "Installed: perl-Net-HTTP-6.17-2.module+el8.3.0+6498+9eecfe51.noarch", "Installed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: perl-Pod-LaTeX-0.61-302.el8.noarch", "Installed: perl-HTTP-Cookies-6.04-2.module+el8.3.0+6498+9eecfe51.noarch", "Installed: urw-base35-standard-symbols-ps-fonts-20170801-10.el8.noarch", "Installed: urw-base35-nimbus-sans-fonts-20170801-10.el8.noarch", "Installed: perl-XML-SAX-1.00-1.el8.noarch", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: urw-base35-c059-fonts-20170801-10.el8.noarch", "Installed: poppler-data-0.4.9-1.el8.noarch", "Installed: perl-libwww-perl-6.34-1.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-XML-NamespaceSupport-1.12-4.el8.noarch", "Installed: perl-Pod-Plainer-1.04-7.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: perl-XML-SAX-Base-1.09-4.el8.noarch", "Installed: redhat-lsb-desktop-4.1-47.el8.x86_64", "Installed: libijs-0.35-5.el8.x86_64", "Installed: perl-WWW-RobotRules-6.02-18.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-submod-multimedia-4.1-47.el8.x86_64", "Installed: adobe-mappings-pdf-20180407-1.el8.noarch", "Installed: cups-1:2.2.6-38.el8.x86_64", "Installed: avahi-0.7-19.el8.x86_64", "Installed: redhat-lsb-cxx-4.1-47.el8.x86_64", "Installed: libXxf86misc-1.0.4-1.el8.x86_64", "Installed: urw-base35-fonts-common-20170801-10.el8.noarch", "Installed: perl-Try-Tiny-0.30-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: urw-base35-z003-fonts-20170801-10.el8.noarch", "Installed: cups-filters-libs-1.20.0-20.el8.x86_64", "Installed: ghostscript-9.25-7.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: libICE-1.0.9-15.el8.x86_64", "In 2021-05-12T11:48:37.789 INFO:teuthology.task.ansible.out:stalled: cups-filesystem-1:2.2.6-38.el8.noarch", "Installed: urw-base35-fonts-20170801-10.el8.noarch", "Installed: urw-base35-nimbus-mono-ps-fonts-20170801-10.el8.noarch", "Installed: cups-ipptool-1:2.2.6-38.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: poppler-utils-0.66.0-27.el8.x86_64", "Installed: perl-File-CheckTree-4.42-303.el8.noarch", "Installed: redhat-lsb-printing-4.1-47.el8.x86_64", "Installed: qemu-img-15:4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64", "Installed: avahi-glib-0.7-19.el8.x86_64", "Installed: libmcpp-2.7.2-20.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: libidn-1.34-5.el8.x86_64", "Installed: urw-base35-bookman-fonts-20170801-10.el8.noarch", "Installed: lcms2-2.9-2.el8.x86_64", "Installed: urw-base35-gothic-fonts-20170801-10.el8.noarch", "Installed: perl-Data-Dump-1.23-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-Class-ISA-0.36-1022.el8.noarch", "Installed: cups-filters-1.20.0-20.el8.x86_64", "Installed: xorg-x11-server-utils-7.7-27.el8.x86_64", "Installed: redhat-lsb-4.1-47.el8.x86_64", "Installed: libgs-9.25-7.el8.x86_64", "Installed: libXmu-1.1.3-1.el8.x86_64", "Installed: mesa-libGLU-9.0.0-15.el8.x86_64", "Installed: perl-Text-Soundex-3.05-8.el8.x86_64", "Installed: urw-base35-nimbus-roman-fonts-20170801-10.el8.noarch", "Installed: libXt-1.1.5-12.el8.x86_64", "Installed: liberation-fonts-common-1:2.00.3-7.el8.noarch", "Installed: adobe-mappings-cmap-20171205-3.el8.noarch", "Installed: liberation-mono-fonts-1:2.00.3-7.el8.noarch", "Installed: libpaper-1.1.24-22.el8.x86_64", "Installed: perl-Module-Pluggable-2:5.2-7.el8.noarch", "Installed: adobe-mappings-cmap-deprecated-20171205-3.el8.noarch", "Installed: perl-File-Listing-6.04-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: google-droid-sans-fonts-20120715-13.el8.noarch", "Installed: perl-Module-Runtime-0.016-2.el8.noarch", "Installed: urw-base35-d050000l-fonts-20170801-10.el8.noarch", "Installed: perl-B-Lint-1.20-11.el8 2021-05-12T11:48:37.791 INFO:teuthology.task.ansible.out:.noarch", "Installed: perl-NTLM-1.09-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: openjpeg2-2.3.1-6.el8.x86_64", "Installed: libSM-1.2.3-1.el8.x86_64", "Installed: perl-XML-LibXML-1:2.0132-2.el8.x86_64", "Installed: perl-HTTP-Negotiate-6.01-19.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-languages-4.1-47.el8.x86_64", "Installed: qpdf-libs-7.1.1-10.el8.x86_64", "Installed: poppler-0.66.0-27.el8.x86_64", "Installed: mcpp-2.7.2-20.el8.x86_64", "Installed: libpng12-1.2.57-5.el8.x86_64", "Installed: jbig2dec-libs-0.14-4.el8_2.x86_64", "Installed: urw-base35-p052-fonts-20170801-10.el8.noarch", "Removed: qemu-img-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2021-05-12T11:48:37.808 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:37 +0000 (0:00:53.745) 0:06:09.421 ********* 2021-05-12T11:48:49.057 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-05-12T11:48:49.059 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:48:49.080 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:49 +0000 (0:00:11.271) 0:06:20.692 ********* 2021-05-12T11:48:51.739 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-05-12T11:48:51.741 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:48:51.762 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:51 +0000 (0:00:02.681) 0:06:23.374 ********* 2021-05-12T11:48:51.830 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-12T11:48:51.832 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:51.853 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:51 +0000 (0:00:00.091) 0:06:23.465 ********* 2021-05-12T11:48:52.180 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-05-12T11:48:52.181 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service sysinit.target dbus.socket polkit.service basic.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service iptables.service ip6tables.service ebtables.service shutdown.target nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-05-12T11:48:52.183 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-05-12T11:48:52.185 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127238", "LimitNPROCSoft": "127238", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127238", "LimitSIGPENDINGSoft": "127238", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-05-12T11:48:52.186 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-05-12T11:48:52.204 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:52 +0000 (0:00:00.350) 0:06:23.816 ********* 2021-05-12T11:48:52.519 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-05-12T11:48:52.521 INFO:teuthology.task.ansible.out:fatal: [smithi191.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-05-12T11:48:52.541 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:52 +0000 (0:00:00.337) 0:06:24.153 ********* 2021-05-12T11:48:53.228 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-05-12T11:48:53.229 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-05-12T11:48:53.249 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:53 +0000 (0:00:00.708) 0:06:24.862 ********* 2021-05-12T11:48:53.317 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-05-12T11:48:53.319 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:53.339 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:53 +0000 (0:00:00.089) 0:06:24.952 ********* 2021-05-12T11:48:53.407 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-05-12T11:48:53.408 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:53.428 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:53 +0000 (0:00:00.089) 0:06:25.041 ********* 2021-05-12T11:48:53.495 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-05-12T11:48:53.497 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:53.519 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:53 +0000 (0:00:00.090) 0:06:25.131 ********* 2021-05-12T11:48:53.586 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-05-12T11:48:53.588 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:53.608 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:53 +0000 (0:00:00.088) 0:06:25.220 ********* 2021-05-12T11:48:53.665 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:53.687 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:53 +0000 (0:00:00.079) 0:06:25.299 ********* 2021-05-12T11:48:53.754 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-05-12T11:48:53.756 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:53.777 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:53 +0000 (0:00:00.089) 0:06:25.389 ********* 2021-05-12T11:48:53.844 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-05-12T11:48:53.867 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:53 +0000 (0:00:00.090) 0:06:25.480 ********* 2021-05-12T11:48:53.934 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-05-12T11:48:53.936 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:53.956 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:53 +0000 (0:00:00.089) 0:06:25.569 ********* 2021-05-12T11:48:54.026 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.048 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.091) 0:06:25.660 ********* 2021-05-12T11:48:54.116 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-05-12T11:48:54.117 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.127 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.149 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.100) 0:06:25.761 ********* 2021-05-12T11:48:54.217 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-05-12T11:48:54.219 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.239 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.090) 0:06:25.851 ********* 2021-05-12T11:48:54.307 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-05-12T11:48:54.329 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.090) 0:06:25.942 ********* 2021-05-12T11:48:54.396 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-05-12T11:48:54.398 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.418 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.088) 0:06:26.030 ********* 2021-05-12T11:48:54.486 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-05-12T11:48:54.488 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.507 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.089) 0:06:26.120 ********* 2021-05-12T11:48:54.585 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-05-12T11:48:54.587 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.610 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.102) 0:06:26.222 ********* 2021-05-12T11:48:54.677 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-12T11:48:54.679 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.699 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.088) 0:06:26.311 ********* 2021-05-12T11:48:54.767 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-05-12T11:48:54.769 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.788 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.089) 0:06:26.401 ********* 2021-05-12T11:48:54.876 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.899 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.110) 0:06:26.511 ********* 2021-05-12T11:48:54.966 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-05-12T11:48:54.968 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:54.988 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:54 +0000 (0:00:00.088) 0:06:26.600 ********* 2021-05-12T11:48:55.055 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-05-12T11:48:55.057 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.077 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.089) 0:06:26.689 ********* 2021-05-12T11:48:55.155 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-05-12T11:48:55.156 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.176 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.099) 0:06:26.789 ********* 2021-05-12T11:48:55.243 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-05-12T11:48:55.245 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.264 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.088) 0:06:26.877 ********* 2021-05-12T11:48:55.334 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-05-12T11:48:55.335 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.355 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.090) 0:06:26.968 ********* 2021-05-12T11:48:55.423 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-05-12T11:48:55.425 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.444 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.089) 0:06:27.057 ********* 2021-05-12T11:48:55.511 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-05-12T11:48:55.513 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.534 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.089) 0:06:27.146 ********* 2021-05-12T11:48:55.602 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-05-12T11:48:55.603 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.623 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.089) 0:06:27.236 ********* 2021-05-12T11:48:55.690 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-05-12T11:48:55.692 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.716 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.091) 0:06:27.327 ********* 2021-05-12T11:48:55.772 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-05-12T11:48:55.774 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.794 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.079) 0:06:27.407 ********* 2021-05-12T11:48:55.852 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.873 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.078) 0:06:27.486 ********* 2021-05-12T11:48:55.940 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-05-12T11:48:55.942 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:55.961 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:55 +0000 (0:00:00.088) 0:06:27.574 ********* 2021-05-12T11:48:56.028 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-05-12T11:48:56.030 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.049 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.088) 0:06:27.662 ********* 2021-05-12T11:48:56.105 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-05-12T11:48:56.107 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.126 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.076) 0:06:27.739 ********* 2021-05-12T11:48:56.182 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.204 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.077) 0:06:27.816 ********* 2021-05-12T11:48:56.270 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-05-12T11:48:56.272 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.291 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.087) 0:06:27.903 ********* 2021-05-12T11:48:56.358 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-05-12T11:48:56.359 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.379 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.087) 0:06:27.991 ********* 2021-05-12T11:48:56.446 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:48:56.448 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.468 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.088) 0:06:28.080 ********* 2021-05-12T11:48:56.535 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-05-12T11:48:56.537 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.557 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.088) 0:06:28.169 ********* 2021-05-12T11:48:56.624 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-12T11:48:56.625 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.645 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.088) 0:06:28.258 ********* 2021-05-12T11:48:56.713 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-12T11:48:56.715 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.737 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.091) 0:06:28.350 ********* 2021-05-12T11:48:56.804 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-05-12T11:48:56.806 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.827 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.090) 0:06:28.440 ********* 2021-05-12T11:48:56.895 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:56.917 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.089) 0:06:28.530 ********* 2021-05-12T11:48:56.984 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-12T11:48:56.985 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.005 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:56 +0000 (0:00:00.088) 0:06:28.618 ********* 2021-05-12T11:48:57.062 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-12T11:48:57.064 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.085 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.079) 0:06:28.698 ********* 2021-05-12T11:48:57.144 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.166 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.081) 0:06:28.779 ********* 2021-05-12T11:48:57.255 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.277 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.111) 0:06:28.890 ********* 2021-05-12T11:48:57.345 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-05-12T11:48:57.347 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.367 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.089) 0:06:28.980 ********* 2021-05-12T11:48:57.435 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-12T11:48:57.437 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.457 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.089) 0:06:29.070 ********* 2021-05-12T11:48:57.525 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-12T11:48:57.526 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.546 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.088) 0:06:29.158 ********* 2021-05-12T11:48:57.614 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-12T11:48:57.616 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.636 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.089) 0:06:29.248 ********* 2021-05-12T11:48:57.692 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-12T11:48:57.694 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.715 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.078) 0:06:29.327 ********* 2021-05-12T11:48:57.783 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-12T11:48:57.785 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.807 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.092) 0:06:29.420 ********* 2021-05-12T11:48:57.865 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.890 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.083) 0:06:29.503 ********* 2021-05-12T11:48:57.958 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-05-12T11:48:57.959 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:57.981 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:57 +0000 (0:00:00.091) 0:06:29.594 ********* 2021-05-12T11:48:58.049 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.071 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.089) 0:06:29.684 ********* 2021-05-12T11:48:58.130 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-05-12T11:48:58.133 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.152 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.080) 0:06:29.764 ********* 2021-05-12T11:48:58.209 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-05-12T11:48:58.210 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.231 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.079) 0:06:29.844 ********* 2021-05-12T11:48:58.299 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-05-12T11:48:58.301 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.320 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.089) 0:06:29.933 ********* 2021-05-12T11:48:58.388 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.410 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.089) 0:06:30.023 ********* 2021-05-12T11:48:58.478 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-05-12T11:48:58.479 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.500 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.089) 0:06:30.113 ********* 2021-05-12T11:48:58.568 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-05-12T11:48:58.569 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.590 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.089) 0:06:30.202 ********* 2021-05-12T11:48:58.657 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-05-12T11:48:58.658 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.680 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.090) 0:06:30.292 ********* 2021-05-12T11:48:58.747 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-05-12T11:48:58.749 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.770 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.090) 0:06:30.382 ********* 2021-05-12T11:48:58.837 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-05-12T11:48:58.839 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.862 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.091) 0:06:30.474 ********* 2021-05-12T11:48:58.929 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:58.950 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:58 +0000 (0:00:00.088) 0:06:30.562 ********* 2021-05-12T11:48:59.017 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-05-12T11:48:59.018 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.039 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:59 +0000 (0:00:00.088) 0:06:30.651 ********* 2021-05-12T11:48:59.106 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:48:59.108 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-05-12T11:48:59.128 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:59 +0000 (0:00:00.089) 0:06:30.740 ********* 2021-05-12T11:48:59.206 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-05-12T11:48:59.208 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.217 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-05-12T11:48:59.218 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.229 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.240 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.241 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.251 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.277 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:59 +0000 (0:00:00.148) 0:06:30.889 ********* 2021-05-12T11:48:59.499 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-05-12T11:48:59.501 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003190", "end": "2021-05-12 11:48:59.481156", "rc": 0, "start": "2021-05-12 11:48:59.477966", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology-lock"]} 2021-05-12T11:48:59.524 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:59 +0000 (0:00:00.247) 0:06:31.137 ********* 2021-05-12T11:48:59.592 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-05-12T11:48:59.593 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.617 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:59 +0000 (0:00:00.092) 0:06:31.229 ********* 2021-05-12T11:48:59.684 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-05-12T11:48:59.686 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.707 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:59 +0000 (0:00:00.089) 0:06:31.319 ********* 2021-05-12T11:48:59.787 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-05-12T11:48:59.789 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.816 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:59 +0000 (0:00:00.107) 0:06:31.427 ********* 2021-05-12T11:48:59.885 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-05-12T11:48:59.887 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:48:59.906 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:48:59 +0000 (0:00:00.091) 0:06:31.519 ********* 2021-05-12T11:49:02.608 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-05-12T11:49:02.610 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:49:02.630 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:02 +0000 (0:00:02.723) 0:06:34.243 ********* 2021-05-12T11:49:02.708 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi191.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930418405376, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227152931, 'block_used': 12942177, 'inode_total': 61054976, 'inode_available': 60986957, 'inode_used': 68019, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227152931, "block_size": 4096, "block_total": 240095108, "block_used": 12942177, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986957, "inode_total": 61054976, "inode_used": 68019, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930418405376, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}} 2021-05-12T11:49:02.730 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:02 +0000 (0:00:00.099) 0:06:34.342 ********* 2021-05-12T11:49:02.952 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-05-12T11:49:02.953 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005132", "end": "2021-05-12 11:49:02.933161", "rc": 0, "start": "2021-05-12 11:49:02.928029", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:02.973 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:02 +0000 (0:00:00.243) 0:06:34.586 ********* 2021-05-12T11:49:03.052 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-05-12T11:49:03.053 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930418405376, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227152931, 'block_used': 12942177, 'inode_total': 61054976, 'inode_available': 60986957, 'inode_used': 68019, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227152931, "block_size": 4096, "block_total": 240095108, "block_used": 12942177, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986957, "inode_total": 61054976, "inode_used": 68019, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930418405376, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:03.075 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:03 +0000 (0:00:00.101) 0:06:34.688 ********* 2021-05-12T11:49:03.143 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-05-12T11:49:03.169 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:03 +0000 (0:00:00.094) 0:06:34.782 ********* 2021-05-12T11:49:04.604 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rpDxpIussoq78oUvdzaUrE0PVYcvssanX'], 'uuids': ['5001c051-b157-426c-9ea1-db4bee279023'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.189096", "end": "2021-05-12 11:49:04.572582", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rpDxpIussoq78oUvdzaUrE0PVYcvssanX"], "labels": [], "masters": [], "uuids": ["5001c051-b157-426c-9ea1-db4bee279023"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:03.383486", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk u 2021-05-12T11:49:04.606 INFO:teuthology.task.ansible.out:sing fdisk or", "other utilities."]} 2021-05-12T11:49:05.891 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-IuIMKH-Gonq-KAgc-Q7YW-I2E1-h586-5lC2DX', 'nvme-INTEL_SSDPEDMD400G4_CVFT6303008M400BGN', 'nvme-nvme.8086-43564654363330333030384d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.108603", "end": "2021-05-12 11:49:05.863022", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-IuIMKH-Gonq-KAgc-Q7YW-I2E1-h586-5lC2DX", "nvme-INTEL_SSDPEDMD400G4_CVFT6303008M400BGN", "nvme-nvme.8086-43564654363330333030384d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:04.754419", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-05-12T11:49:05.893 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:49:07.124 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rupHFU9xttTJ2QZNwl0dHIwmtZUB5bols'], 'uuids': ['c094cd17-7b53-4fe1-a974-d50b1ec5af6c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.052429", "end": "2021-05-12 11:49:07.098487", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rupHFU9xttTJ2QZNwl0dHIwmtZUB5bols"], "labels": [], "masters": [], "uuids": ["c094cd17-7b53-4fe1-a974-d50b1ec5af6c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:06.046058", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:49:08.366 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5r9fFcluBLMVh87Ck727SnnZtijtDYGP8G'], 'uuids': ['a4849639-68dc-4a10-9d38-a696587bd7cd'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.052964", "end": "2021-05-12 11:49:08.333206", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5r9fFcluBLMVh87Ck727SnnZtijtDYGP8G"], "labels": [], "masters": [], "uuids": ["a4849639-68dc-4a10-9d38-a696587bd7cd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:07.280242", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:49:09.607 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rjmqydIm9A7A5daCMjxAeJc75Nhv3pmH3'], 'uuids': ['993e834a-c98f-4872-8e69-8273ebc77aab'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.064147", "end": "2021-05-12 11:49:09.574317", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rjmqydIm9A7A5daCMjxAeJc75Nhv3pmH3"], "labels": [], "masters": [], "uuids": ["993e834a-c98f-4872-8e69-8273ebc77aab"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:08.510170", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:49:09.609 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XMDN', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XMDN', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XMDN', 'scsi-35000c50092933758', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XMDN', 'wwn-0x5000c50092933758'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XMDN', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092933758', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XMDN-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XMDN-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XMDN-part1', 'scsi-35000c50092933758-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XMDN-part1', 'wwn-0x5000c50092933758-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XMDN", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XMDN", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XMDN", "scsi-35000c50092933758", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XMDN", "wwn-0x5000c50092933758"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XMDN-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XMDN-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-05-12T11:49:09.610 INFO:teuthology.task.ansible.out:Z1W5XMDN-part1", "scsi-35000c50092933758-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XMDN-part1", "wwn-0x5000c50092933758-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XMDN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092933758"}}, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:10.847 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5r1cfSmmic9hdBnDdj1xGFeM14gYVvgwYc'], 'uuids': ['707a03f1-bce2-4418-b8a7-9e21b3d925ad'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.053550", "end": "2021-05-12 11:49:10.816001", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5r1cfSmmic9hdBnDdj1xGFeM14gYVvgwYc"], "labels": [], "masters": [], "uuids": ["707a03f1-bce2-4418-b8a7-9e21b3d925ad"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:09.762451", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:49:10.878 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:10 +0000 (0:00:07.708) 0:06:42.490 ********* 2021-05-12T11:49:11.269 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-05-12T11:49:11.271 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rpDxpIussoq78oUvdzaUrE0PVYcvssanX'], 'uuids': ['5001c051-b157-426c-9ea1-db4bee279023'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.140585", "end": "2021-05-12 11:49:11.241502", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rpDxpIussoq78oUvdzaUrE0PVYcvssanX"], "labels": [], "masters": [], "uuids": ["5001c051-b157-426c-9ea1-db4bee279023"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:11.100917", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137823 s, 837 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137823 s, 837 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:11.479 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-IuIMKH-Gonq-KAgc-Q7YW-I2E1-h586-5lC2DX', 'nvme-INTEL_SSDPEDMD400G4_CVFT6303008M400BGN', 'nvme-nvme.8086-43564654363330333030384d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.036406", "end": "2021-05-12 11:49:11.451166", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-IuIMKH-Gonq-KAgc-Q7YW-I2E1-h586-5lC2DX", "nvme-INTEL_SSDPEDMD400G4_CVFT6303008M400BGN", "nvme-nvme.8086-43564654363330333030384d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:11.414760", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0337174 s, 3.4 GB/s", "std 2021-05-12T11:49:11.480 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0337174 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:11.792 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rupHFU9xttTJ2QZNwl0dHIwmtZUB5bols'], 'uuids': ['c094cd17-7b53-4fe1-a974-d50b1ec5af6c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.141367", "end": "2021-05-12 11:49:11.765962", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rupHFU9xttTJ2QZNwl0dHIwmtZUB5bols"], "labels": [], "masters": [], "uuids": ["c094cd17-7b53-4fe1-a974-d50b1ec5af6c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:11.624595", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138677 s, 832 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138677 s, 832 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:12.104 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5r9fFcluBLMVh87Ck727SnnZtijtDYGP8G'], 'uuids': ['a4849639-68dc-4a10-9d38-a696587bd7cd'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.139944", "end": "2021-05-12 11:49:12.073550", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5r9fFcluBLMVh87Ck727SnnZtijtDYGP8G"], "labels": [], "masters": [], "uuids": ["a4849639-68dc-4a10-9d38-a696587bd7cd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:11.933606", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137354 s, 840 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137354 s, 840 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:12.415 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rjmqydIm9A7A5daCMjxAeJc75Nhv3pmH3'], 'uuids': ['993e834a-c98f-4872-8e69-8273ebc77aab'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.153794", "end": "2021-05-12 11:49:12.393566", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5rjmqydIm9A7A5daCMjxAeJc75Nhv3pmH3"], "labels": [], "masters": [], "uuids": ["993e834a-c98f-4872-8e69-8273ebc77aab"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:12.239772", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151203 s, 763 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151203 s, 763 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:12.428 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XMDN', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XMDN', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XMDN', 'scsi-35000c50092933758', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XMDN', 'wwn-0x5000c50092933758'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XMDN', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092933758', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XMDN-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XMDN-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XMDN-part1', 'scsi-35000c50092933758-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XMDN-part1', 'wwn-0x5000c50092933758-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XMDN", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XMDN", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XMDN", "scsi-35000c50092933758", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XMDN", "wwn-0x5000c50092933758"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XMDN-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XMDN-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-05-12T11:49:12.430 INFO:teuthology.task.ansible.out:Z1W5XMDN-part1", "scsi-35000c50092933758-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XMDN-part1", "wwn-0x5000c50092933758-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XMDN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092933758"}}, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:12.732 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5r1cfSmmic9hdBnDdj1xGFeM14gYVvgwYc'], 'uuids': ['707a03f1-bce2-4418-b8a7-9e21b3d925ad'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.142303", "end": "2021-05-12 11:49:12.707749", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-N39fMRyRJGTiiUhugJNknDKMcjt8tR5r1cfSmmic9hdBnDdj1xGFeM14gYVvgwYc"], "labels": [], "masters": [], "uuids": ["707a03f1-bce2-4418-b8a7-9e21b3d925ad"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:49:12.565446", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139686 s, 826 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139686 s, 826 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:12.759 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:12 +0000 (0:00:01.881) 0:06:44.372 ********* 2021-05-12T11:49:13.027 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-05-12T11:49:13.029 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.051261", "end": "2021-05-12 11:49:12.965867", "rc": 0, "start": "2021-05-12 11:49:12.914606", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:13.049 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.289) 0:06:44.661 ********* 2021-05-12T11:49:13.285 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-05-12T11:49:13.287 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.057808", "end": "2021-05-12 11:49:13.258632", "rc": 0, "start": "2021-05-12 11:49:13.200824", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:13.307 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.257) 0:06:44.919 ********* 2021-05-12T11:49:13.375 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-05-12T11:49:13.434 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.127) 0:06:45.047 ********* 2021-05-12T11:49:13.462 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-05-12T11:49:13.464 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:13.485 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.050) 0:06:45.098 ********* 2021-05-12T11:49:13.512 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-05-12T11:49:13.513 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:13.534 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.048) 0:06:45.147 ********* 2021-05-12T11:49:13.563 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-05-12T11:49:13.565 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:13.587 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.052) 0:06:45.199 ********* 2021-05-12T11:49:13.613 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-05-12T11:49:13.615 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:13.635 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.048) 0:06:45.247 ********* 2021-05-12T11:49:13.828 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-05-12T11:49:13.830 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:49:13.850 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.215) 0:06:45.463 ********* 2021-05-12T11:49:13.898 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-05-12T11:49:13.900 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930418405376, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227152931, 'block_used': 12942177, 'inode_total': 61054976, 'inode_available': 60986957, 'inode_used': 68019, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227152931, "block_size": 4096, "block_total": 240095108, "block_used": 12942177, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986957, "inode_total": 61054976, "inode_used": 68019, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930418405376, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:13.922 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.071) 0:06:45.534 ********* 2021-05-12T11:49:13.949 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-05-12T11:49:13.951 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:13.970 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:13 +0000 (0:00:00.048) 0:06:45.583 ********* 2021-05-12T11:49:14.009 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:49:14.011 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:14.030 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:14 +0000 (0:00:00.060) 0:06:45.643 ********* 2021-05-12T11:49:14.057 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:49:14.059 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:14.079 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:14 +0000 (0:00:00.048) 0:06:45.691 ********* 2021-05-12T11:49:14.105 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-05-12T11:49:14.108 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:14.127 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:14 +0000 (0:00:00.048) 0:06:45.740 ********* 2021-05-12T11:49:14.155 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:14.178 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:14 +0000 (0:00:00.049) 0:06:45.790 ********* 2021-05-12T11:49:14.205 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-05-12T11:49:14.206 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:14.230 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:14 +0000 (0:00:00.052) 0:06:45.842 ********* 2021-05-12T11:49:14.943 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-05-12T11:49:14.946 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-05-12T11:49:14.975 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:14 +0000 (0:00:00.745) 0:06:46.588 ********* 2021-05-12T11:49:15.626 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-05-12T11:49:15.628 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:49:15.960 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:49:16.282 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:49:16.610 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:49:16.921 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:49:16.955 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:16 +0000 (0:00:01.979) 0:06:48.567 ********* 2021-05-12T11:49:17.136 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-05-12T11:49:17.138 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-05-12T11:49:17.158 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:17 +0000 (0:00:00.203) 0:06:48.771 ********* 2021-05-12T11:49:17.360 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-05-12T11:49:17.361 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T11:49:17.528 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T11:49:17.705 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T11:49:17.873 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T11:49:17.883 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:17.908 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:17 +0000 (0:00:00.749) 0:06:49.520 ********* 2021-05-12T11:49:17.935 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:17.956 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:17 +0000 (0:00:00.047) 0:06:49.568 ********* 2021-05-12T11:49:17.982 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:18.004 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:17 +0000 (0:00:00.048) 0:06:49.617 ********* 2021-05-12T11:49:18.196 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-05-12T11:49:18.198 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T11:49:18.218 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:18 +0000 (0:00:00.213) 0:06:49.830 ********* 2021-05-12T11:49:18.254 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-05-12T11:49:18.256 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:18.278 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:18 +0000 (0:00:00.059) 0:06:49.890 ********* 2021-05-12T11:49:18.822 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-05-12T11:49:18.824 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true} 2021-05-12T11:49:18.845 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:18 +0000 (0:00:00.567) 0:06:50.458 ********* 2021-05-12T11:49:20.453 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-05-12T11:49:20.454 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-05-12T11:49:20.474 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:01.628) 0:06:52.087 ********* 2021-05-12T11:49:20.501 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-05-12T11:49:20.503 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.523 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.048) 0:06:52.136 ********* 2021-05-12T11:49:20.550 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-05-12T11:49:20.552 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.572 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.048) 0:06:52.184 ********* 2021-05-12T11:49:20.598 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-12T11:49:20.600 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.619 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.047) 0:06:52.232 ********* 2021-05-12T11:49:20.646 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-12T11:49:20.648 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.667 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.047) 0:06:52.280 ********* 2021-05-12T11:49:20.693 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-05-12T11:49:20.695 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.714 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.046) 0:06:52.327 ********* 2021-05-12T11:49:20.740 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-05-12T11:49:20.742 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.761 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.047) 0:06:52.374 ********* 2021-05-12T11:49:20.788 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-05-12T11:49:20.789 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.810 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.048) 0:06:52.422 ********* 2021-05-12T11:49:20.836 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-05-12T11:49:20.838 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.858 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.048) 0:06:52.471 ********* 2021-05-12T11:49:20.885 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-05-12T11:49:20.887 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.914 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.056) 0:06:52.527 ********* 2021-05-12T11:49:20.941 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-05-12T11:49:20.943 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:20.963 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:20 +0000 (0:00:00.048) 0:06:52.575 ********* 2021-05-12T11:49:20.990 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-05-12T11:49:20.992 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:21.011 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:21 +0000 (0:00:00.047) 0:06:52.623 ********* 2021-05-12T11:49:21.037 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:21.061 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:21 +0000 (0:00:00.050) 0:06:52.673 ********* 2021-05-12T11:49:21.087 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-05-12T11:49:21.089 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:21.109 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:21 +0000 (0:00:00.047) 0:06:52.721 ********* 2021-05-12T11:49:21.135 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-05-12T11:49:21.137 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:21.157 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:21 +0000 (0:00:00.048) 0:06:52.770 ********* 2021-05-12T11:49:21.183 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-05-12T11:49:21.185 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:21.205 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:21 +0000 (0:00:00.047) 0:06:52.817 ********* 2021-05-12T11:49:21.668 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-05-12T11:49:21.669 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-05-12T11:49:21.690 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:21 +0000 (0:00:00.485) 0:06:53.303 ********* 2021-05-12T11:49:21.990 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-05-12T11:49:21.992 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-05-12 11:41:35 UTC", "ActiveEnterTimestampMonotonic": "16197287", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service -.mount sysinit.target tmp.mount ntpd.service basic.target systemd-tmpfiles-setup.service systemd-journald.socket sntp.service system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-05-12 11:41:34 UTC", "AssertTimestampMonotonic": "14982618", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-05-12 11:41:34 UTC", "ConditionTimestampMonotonic": "14982589", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service ntpd.service shutdown.target", "ControlGro 2021-05-12T11:49:21.994 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "897", "ExecMainStartTimestamp": "Wed 2021-05-12 11:41:35 UTC", "ExecMainStartTimestampMonotonic": "16094888", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-05-12 11:41:34 UTC", "InactiveExitTimestampMonotonic": "15120116", "InvocationID": "f6039b12f82b46cc94fee5b43d9c4122", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-05-12T11:49:21.995 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127238", "LimitNPROCSoft": "127238", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127238", "LimitSIGPENDINGSoft": "127238", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "897", "MemoryAccounting": "yes", "MemoryCurrent": "3166208", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-05-12T11:49:21.997 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-05-12 11:41:35 UTC", "StateChangeTimestampMonotonic": "16197287", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-05-12 11:41:35 UTC", "WatchdogTimestampMonotonic": "16094903", "WatchdogUSec": "0"}} 2021-05-12T11:49:22.024 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:22 +0000 (0:00:00.334) 0:06:53.637 ********* 2021-05-12T11:49:22.052 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:49:22.054 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-05-12T11:49:22.074 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:22 +0000 (0:00:00.050) 0:06:53.687 ********* 2021-05-12T11:49:22.112 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:49:22.114 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-05-12T11:49:22.134 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:22 +0000 (0:00:00.059) 0:06:53.746 ********* 2021-05-12T11:49:22.160 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-05-12T11:49:22.161 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:22.182 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:22 +0000 (0:00:00.048) 0:06:53.795 ********* 2021-05-12T11:49:22.209 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-05-12T11:49:22.211 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:22.231 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:22 +0000 (0:00:00.049) 0:06:53.844 ********* 2021-05-12T11:49:22.259 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:22.280 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:22 +0000 (0:00:00.048) 0:06:53.892 ********* 2021-05-12T11:49:22.472 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-05-12T11:49:22.474 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-05-12T11:49:22.496 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:22 +0000 (0:00:00.216) 0:06:54.109 ********* 2021-05-12T11:49:22.970 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-05-12T11:49:22.971 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-05-12T11:49:22.996 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:22 +0000 (0:00:00.499) 0:06:54.609 ********* 2021-05-12T11:49:24.305 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-05-12T11:49:24.307 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-05-12T11:49:24.327 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:24 +0000 (0:00:01.331) 0:06:55.940 ********* 2021-05-12T11:49:24.354 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-05-12T11:49:24.356 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:24.378 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:24 +0000 (0:00:00.050) 0:06:55.990 ********* 2021-05-12T11:49:24.404 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-05-12T11:49:24.405 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-05-12T11:49:24.427 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:24 +0000 (0:00:00.049) 0:06:56.039 ********* 2021-05-12T11:49:24.599 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-05-12T11:49:24.600 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-05-12T11:49:24.622 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:24 +0000 (0:00:00.194) 0:06:56.234 ********* 2021-05-12T11:49:24.649 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-05-12T11:49:24.651 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:24.676 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:24 +0000 (0:00:00.053) 0:06:56.288 ********* 2021-05-12T11:49:24.702 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-05-12T11:49:24.724 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:24 +0000 (0:00:00.048) 0:06:56.337 ********* 2021-05-12T11:49:24.752 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-05-12T11:49:24.754 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:24.775 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:24 +0000 (0:00:00.051) 0:06:56.388 ********* 2021-05-12T11:49:24.959 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-05-12T11:49:24.961 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-05-12T11:49:24.983 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:24 +0000 (0:00:00.207) 0:06:56.596 ********* 2021-05-12T11:49:25.187 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi191.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-05-12T11:49:25.208 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.225) 0:06:56.821 ********* 2021-05-12T11:49:25.234 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-05-12T11:49:25.236 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.256 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.047) 0:06:56.869 ********* 2021-05-12T11:49:25.283 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi191.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.294 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.296 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.305 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.306 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.328 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.071) 0:06:56.940 ********* 2021-05-12T11:49:25.364 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-05-12T11:49:25.366 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.376 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.378 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi191.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.400 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.072) 0:06:57.012 ********* 2021-05-12T11:49:25.426 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:49:25.427 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.449 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.049) 0:06:57.061 ********* 2021-05-12T11:49:25.475 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-05-12T11:49:25.477 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.497 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.048) 0:06:57.110 ********* 2021-05-12T11:49:25.523 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.548 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.050) 0:06:57.160 ********* 2021-05-12T11:49:25.574 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-05-12T11:49:25.576 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.596 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.048) 0:06:57.209 ********* 2021-05-12T11:49:25.622 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-05-12T11:49:25.624 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.647 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.050) 0:06:57.259 ********* 2021-05-12T11:49:25.673 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-05-12T11:49:25.675 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.697 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.050) 0:06:57.309 ********* 2021-05-12T11:49:25.724 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-05-12T11:49:25.725 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.747 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.050) 0:06:57.360 ********* 2021-05-12T11:49:25.774 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-05-12T11:49:25.775 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:25.795 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:25 +0000 (0:00:00.047) 0:06:57.407 ********* 2021-05-12T11:49:25.999 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi191.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005190", "end": "2021-05-12 11:49:25.966445", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-05-12 11:49:25.961255", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:26.165 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002969", "end": "2021-05-12 11:49:26.135140", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-05-12 11:49:26.132171", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:26.332 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002747", "end": "2021-05-12 11:49:26.301731", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-05-12 11:49:26.298984", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:26.502 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003255", "end": "2021-05-12 11:49:26.468194", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-05-12 11:49:26.464939", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:26.524 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:26 +0000 (0:00:00.729) 0:06:58.137 ********* 2021-05-12T11:49:26.550 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-05-12T11:49:26.572 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:26 +0000 (0:00:00.048) 0:06:58.185 ********* 2021-05-12T11:49:26.754 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-05-12T11:49:26.756 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-12T11:49:26.824 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-05-12T11:49:26.840 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:26 +0000 (0:00:00.268) 0:06:58.453 ********* 2021-05-12T11:49:27.498 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi191.front.sepia.ceph.com] 2021-05-12T11:49:27.554 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:27 +0000 (0:00:00.713) 0:06:59.166 ********* 2021-05-12T11:49:27.582 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-12T11:49:27.584 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:27.605 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:27 +0000 (0:00:00.052) 0:06:59.218 ********* 2021-05-12T11:49:27.642 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-12T11:49:27.643 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-05-12T11:49:27.663 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:27 +0000 (0:00:00.057) 0:06:59.276 ********* 2021-05-12T11:49:27.699 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-05-12T11:49:27.701 INFO:teuthology.task.ansible.out:ok: [smithi191.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-05-12T11:49:27.721 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:27 +0000 (0:00:00.058) 0:06:59.334 ********* 2021-05-12T11:49:46.150 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-05-12T11:49:46.152 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x86_64", "Installed: podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526.noarch", "Installed: podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64", "Installed: podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64", "Removed: podman-catatonit-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64", "Removed: podman-docker-2.0.5-5.module+el8.3.0+8221+97165c3f.noarch", "Removed: runc-1.0.0-68.rc92.module+el8.3.0+8221+97165c3f.x86_64", "Removed: podman-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64"]} 2021-05-12T11:49:46.172 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:46 +0000 (0:00:18.451) 0:07:17.785 ********* 2021-05-12T11:49:46.200 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi191.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-05-12T11:49:46.222 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:46 +0000 (0:00:00.049) 0:07:17.834 ********* 2021-05-12T11:49:46.249 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-12T11:49:46.251 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:46.270 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:46 +0000 (0:00:00.048) 0:07:17.883 ********* 2021-05-12T11:49:46.486 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-05-12T11:49:46.487 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T11:49:46.663 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T11:49:46.689 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:46 +0000 (0:00:00.418) 0:07:18.302 ********* 2021-05-12T11:49:47.193 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-05-12T11:49:47.195 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620820186.7544026-13695-24470113874704/source", "state": "file", "uid": 0} 2021-05-12T11:49:47.652 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620820187.2211676-13695-9375774394104/source", "state": "file", "uid": 0} 2021-05-12T11:49:47.675 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:47 +0000 (0:00:00.985) 0:07:19.288 ********* 2021-05-12T11:49:51.117 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-05-12T11:49:51.119 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6przvqpa-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6przvqpa-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-05-12T11:49:51.121 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-05-12T11:49:51.140 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:51 +0000 (0:00:03.464) 0:07:22.752 ********* 2021-05-12T11:49:51.177 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-05-12T11:49:51.178 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:51.201 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:51 +0000 (0:00:00.061) 0:07:22.814 ********* 2021-05-12T11:49:51.228 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-05-12T11:49:51.230 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:51.252 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:51 +0000 (0:00:00.050) 0:07:22.864 ********* 2021-05-12T11:49:51.595 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-05-12T11:49:51.597 INFO:teuthology.task.ansible.out:changed: [smithi191.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.158776", "end": "2021-05-12 11:49:51.573723", "rc": 0, "start": "2021-05-12 11:49:51.414947", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:49:51.617 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:49:51 +0000 (0:00:00.365) 0:07:23.229 ********* 2021-05-12T11:49:51.643 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-05-12T11:49:51.646 INFO:teuthology.task.ansible.out:skipping: [smithi191.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:49:51.715 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched PLAY [paddles] ***************************************************************** skipping: no hosts matched PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi191.front.sepia.ceph.com : ok=137 changed=47 unreachable=0 failed=0 skipped=175 rescued=0 ignored=4 Wednesday 12 May 2021 11:49:51 +0000 (0:00:00.097) 0:07:23.327 ********* =============================================================================== 2021-05-12T11:49:51.716 INFO:teuthology.task.ansible.out:testnode : Install packages -------------------------------------------- 53.75s common : nrpe - Ensure SELinux policy is up to date -------------------- 49.66s Create all admin users with sudo access. ------------------------------- 33.60s users : Update authorized_keys using the keys repo --------------------- 31.81s testnode : Ensure ceph packages are not present. ----------------------- 23.27s 2021-05-12T11:49:51.716 INFO:teuthology.task.ansible.out:testnode : Ensure ceph dependency packages are not present. ------------ 23.07s container-host : Install container packages ---------------------------- 18.45s common : Install nrpe package and dependencies (RHEL/CentOS) ----------- 17.32s common : nrpe - Remove SELinux policy package -------------------------- 16.48s common : nrpe - Load SELinux policy package ---------------------------- 16.47s common : Register with subscription-manager. --------------------------- 15.78s Remove revoked users --------------------------------------------------- 13.14s common : Install Kerberos Packages (RedHat) ---------------------------- 11.56s testnode : Install epel packages --------------------------------------- 11.27s common : nrpe - Install SELinux dependencies --------------------------- 11.22s testnode : Zap all non-root disks --------------------------------------- 7.71s container-host : Install registries-conf-ctl ---------------------------- 3.46s common : Enable necessary rhsm repos ------------------------------------ 3.37s testnode : Make sure rpm dependencies are installed --------------------- 2.72s testnode : Remove packages ---------------------------------------------- 2.68s 2021-05-12T11:49:52.101 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2021-05-12T11:49:52.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:49:52.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:49:52.179 DEBUG:teuthology.orchestra.run.smithi191:> true 2021-05-12T11:49:52.270 INFO:teuthology.run_tasks:Running task clock... 2021-05-12T11:49:52.284 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-05-12T11:49:52.284 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-05-12T11:49:52.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-05-12T11:49:52.341 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-05-12T11:49:52.355 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-05-12T11:49:52.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2021-05-12T11:49:52.399 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2021-05-12T11:49:52.412 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-05-12T11:49:52.426 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-05-12T11:49:52.479 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2021-05-12T11:49:52.482 INFO:teuthology.orchestra.run.smithi191.stdout:210 Number of sources = 4 2021-05-12T11:49:52.482 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-05-12T11:49:52.483 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2021-05-12T11:49:52.483 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T11:49:52.483 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T11:49:52.483 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T11:49:52.483 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T11:49:52.484 INFO:teuthology.run_tasks:Running task install... 2021-05-12T11:49:52.495 DEBUG:teuthology.task.install:project ceph 2021-05-12T11:49:52.496 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '36c2cf2fb22a9790b2c097e158985773b35f1740'}} 2021-05-12T11:49:52.496 DEBUG:teuthology.task.install:config {'sha1': '36c2cf2fb22a9790b2c097e158985773b35f1740'} 2021-05-12T11:49:52.496 INFO:teuthology.task.install:Using flavor: basic 2021-05-12T11:49:52.503 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-05-12T11:49:52.503 INFO:teuthology.task.install:extra packages: [] 2021-05-12T11:49:52.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo lsb_release -is 2021-05-12T11:49:52.564 INFO:teuthology.orchestra.run.smithi191.stdout:RedHatEnterprise 2021-05-12T11:49:52.564 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-05-12T11:49:52.565 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '36c2cf2fb22a9790b2c097e158985773b35f1740', 'tag': None, 'wait_for_package': False} 2021-05-12T11:49:52.565 DEBUG:teuthology.orchestra.run.smithi191:> sudo lsb_release -is 2021-05-12T11:49:52.640 INFO:teuthology.orchestra.run.smithi191.stdout:RedHatEnterprise 2021-05-12T11:49:52.641 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-05-12T11:49:52.642 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=36c2cf2fb22a9790b2c097e158985773b35f1740 2021-05-12T11:49:52.793 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1746/36c2cf2fb22a9790b2c097e158985773b35f1740/centos/8/flavors/default/ 2021-05-12T11:49:52.794 INFO:teuthology.task.install.rpm:Package version is 17.0.0-4087.g36c2cf2f 2021-05-12T11:49:52.913 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1746/36c2cf2fb22a9790b2c097e158985773b35f1740/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1746/36c2cf2fb22a9790b2c097e158985773b35f1740/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1746/36c2cf2fb22a9790b2c097e158985773b35f1740/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-05-12T11:49:52.914 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2021-05-12T11:49:52.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-05-12T11:49:52.939 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-05-12T11:49:52.940 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/36c2cf2fb22a9790b2c097e158985773b35f1740/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-05-12T11:49:53.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-05-12T11:49:53.080 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-05-12T11:49:53.143 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2021-05-12T11:49:53.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2021-05-12T11:49:53.447 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:49:53.693 INFO:teuthology.orchestra.run.smithi191.stdout:137 files removed 2021-05-12T11:49:53.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw 2021-05-12T11:49:53.993 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:49:55.042 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 111 kB/s | 77 kB 00:00 2021-05-12T11:49:55.360 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 54 kB/s | 16 kB 00:00 2021-05-12T11:49:55.583 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 7.2 kB/s | 1.5 kB 00:00 2021-05-12T11:49:56.624 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 9.4 MB 00:01 2021-05-12T11:49:58.482 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 5.3 MB/s | 24 kB 00:00 2021-05-12T11:49:58.757 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 118 MB/s | 26 MB 00:00 2021-05-12T11:50:02.984 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 130 MB/s | 29 MB 00:00 2021-05-12T11:50:09.571 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:50:09.609 INFO:teuthology.orchestra.run.smithi191.stdout:==================================================================================================== 2021-05-12T11:50:09.610 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2021-05-12T11:50:09.610 INFO:teuthology.orchestra.run.smithi191.stdout:==================================================================================================== 2021-05-12T11:50:09.610 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:50:09.611 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 9.9 M 2021-05-12T11:50:09.611 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2021-05-12T11:50:09.611 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.3 M 2021-05-12T11:50:09.611 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.8 M 2021-05-12T11:50:09.611 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2021-05-12T11:50:09.612 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 5.4 M 2021-05-12T11:50:09.612 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 19 M 2021-05-12T11:50:09.612 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 23 k 2021-05-12T11:50:09.612 INFO:teuthology.orchestra.run.smithi191.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-05-12T11:50:09.613 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-05-12T11:50:09.613 INFO:teuthology.orchestra.run.smithi191.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-05-12T11:50:09.613 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 756 k 2021-05-12T11:50:09.613 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-05-12T11:50:09.613 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.9.0-2.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-05-12T11:50:09.614 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 458 k 2021-05-12T11:50:09.614 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-05-12T11:50:09.615 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.8 M 2021-05-12T11:50:09.615 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-05-12T11:50:09.615 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-05-12T11:50:09.615 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 42 k 2021-05-12T11:50:09.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 77 k 2021-05-12T11:50:09.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 177 k 2021-05-12T11:50:09.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-05-12T11:50:09.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 364 k 2021-05-12T11:50:09.616 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 345 k 2021-05-12T11:50:09.617 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 109 k 2021-05-12T11:50:09.617 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:50:09.617 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:50:09.617 INFO:teuthology.orchestra.run.smithi191.stdout:==================================================================================================== 2021-05-12T11:50:09.618 INFO:teuthology.orchestra.run.smithi191.stdout:Install 22 Packages 2021-05-12T11:50:09.618 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2021-05-12T11:50:09.618 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:50:09.619 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 49 M 2021-05-12T11:50:09.620 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:50:10.311 INFO:teuthology.orchestra.run.smithi191.stdout:(1/24): ceph-base-17.0.0-4087.g36c2cf2f.el8.x86 7.7 MB/s | 5.4 MB 00:00 2021-05-12T11:50:10.368 INFO:teuthology.orchestra.run.smithi191.stdout:(2/24): ceph-selinux-17.0.0-4087.g36c2cf2f.el8. 410 kB/s | 23 kB 00:00 2021-05-12T11:50:10.513 INFO:teuthology.orchestra.run.smithi191.stdout:(3/24): libcephfs2-17.0.0-4087.g36c2cf2f.el8.x8 5.1 MB/s | 756 kB 00:00 2021-05-12T11:50:10.584 INFO:teuthology.orchestra.run.smithi191.stdout:(4/24): libradosstriper1-17.0.0-4087.g36c2cf2f. 6.3 MB/s | 458 kB 00:00 2021-05-12T11:50:10.996 INFO:teuthology.orchestra.run.smithi191.stdout:(5/24): librgw2-17.0.0-4087.g36c2cf2f.el8.x86_6 9.3 MB/s | 3.8 MB 00:00 2021-05-12T11:50:11.045 INFO:teuthology.orchestra.run.smithi191.stdout:(6/24): python3-ceph-argparse-17.0.0-4087.g36c2 862 kB/s | 42 kB 00:00 2021-05-12T11:50:11.081 INFO:teuthology.orchestra.run.smithi191.stdout:(7/24): ceph-radosgw-17.0.0-4087.g36c2cf2f.el8. 6.7 MB/s | 9.9 MB 00:01 2021-05-12T11:50:11.088 INFO:teuthology.orchestra.run.smithi191.stdout:(8/24): python3-ceph-common-17.0.0-4087.g36c2cf 1.8 MB/s | 77 kB 00:00 2021-05-12T11:50:11.125 INFO:teuthology.orchestra.run.smithi191.stdout:(9/24): python3-cephfs-17.0.0-4087.g36c2cf2f.el 3.9 MB/s | 177 kB 00:00 2021-05-12T11:50:11.143 INFO:teuthology.orchestra.run.smithi191.stdout:(10/24): python3-rados-17.0.0-4087.g36c2cf2f.el 6.5 MB/s | 364 kB 00:00 2021-05-12T11:50:11.295 INFO:teuthology.orchestra.run.smithi191.stdout:(11/24): ceph-common-17.0.0-4087.g36c2cf2f.el8. 11 MB/s | 19 MB 00:01 2021-05-12T11:50:11.296 INFO:teuthology.orchestra.run.smithi191.stdout:(12/24): python3-rgw-17.0.0-4087.g36c2cf2f.el8. 714 kB/s | 109 kB 00:00 2021-05-12T11:50:11.298 INFO:teuthology.orchestra.run.smithi191.stdout:(13/24): python3-rbd-17.0.0-4087.g36c2cf2f.el8. 2.0 MB/s | 345 kB 00:00 2021-05-12T11:50:11.516 INFO:teuthology.orchestra.run.smithi191.stdout:(14/24): fmt-6.2.1-1.el8.x86_64.rpm 527 kB/s | 116 kB 00:00 2021-05-12T11:50:11.538 INFO:teuthology.orchestra.run.smithi191.stdout:(15/24): liboath-2.6.2-3.el8.x86_64.rpm 2.6 MB/s | 59 kB 00:00 2021-05-12T11:50:11.555 INFO:teuthology.orchestra.run.smithi191.stdout:(16/24): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2021-05-12T11:50:11.564 INFO:teuthology.orchestra.run.smithi191.stdout:(17/24): libunwind-1.3.1-3.el8.x86_64.rpm 2.8 MB/s | 75 kB 00:00 2021-05-12T11:50:11.578 INFO:teuthology.orchestra.run.smithi191.stdout:(18/24): leveldb-1.22-1.el8.x86_64.rpm 647 kB/s | 181 kB 00:00 2021-05-12T11:50:11.595 INFO:teuthology.orchestra.run.smithi191.stdout:(19/24): librdkafka-0.11.4-1.el8.x86_64.rpm 8.5 MB/s | 353 kB 00:00 2021-05-12T11:50:11.616 INFO:teuthology.orchestra.run.smithi191.stdout:(20/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.9 MB/s | 259 kB 00:00 2021-05-12T11:50:11.621 INFO:teuthology.orchestra.run.smithi191.stdout:(21/24): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2021-05-12T11:50:11.627 INFO:teuthology.orchestra.run.smithi191.stdout:(22/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 2.0 MB/s | 47 kB 00:00 2021-05-12T11:50:11.872 INFO:teuthology.orchestra.run.smithi191.stdout:(23/24): librados2-17.0.0-4087.g36c2cf2f.el8.x8 13 MB/s | 3.3 MB 00:00 2021-05-12T11:50:11.892 INFO:teuthology.orchestra.run.smithi191.stdout:(24/24): librbd1-17.0.0-4087.g36c2cf2f.el8.x86_ 14 MB/s | 3.8 MB 00:00 2021-05-12T11:50:11.895 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:50:11.895 INFO:teuthology.orchestra.run.smithi191.stdout:Total 22 MB/s | 49 MB 00:02 2021-05-12T11:50:11.899 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:50:11.943 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:50:11.943 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:50:12.325 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:50:12.330 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:50:13.387 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:50:13.624 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-05-12T11:50:13.820 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/26 2021-05-12T11:50:14.214 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/26 2021-05-12T11:50:14.411 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 3/26 2021-05-12T11:50:14.512 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 3/26 2021-05-12T11:50:14.734 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:17.0.0-4087.g36c2cf2f.el8.x86_64 4/26 2021-05-12T11:50:14.968 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/26 2021-05-12T11:50:15.153 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 6/26 2021-05-12T11:50:15.639 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 6/26 2021-05-12T11:50:15.831 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/26 2021-05-12T11:50:15.921 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/26 2021-05-12T11:50:16.152 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/26 2021-05-12T11:50:16.361 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-12T11:50:16.792 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-12T11:50:16.978 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 10/26 2021-05-12T11:50:17.063 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 10/26 2021-05-12T11:50:17.259 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-05-12T11:50:17.387 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:17.0.0-4087.g36c2cf2f.el8. 12/26 2021-05-12T11:50:17.614 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:17.0.0-4087.g36c2cf2f.el8.x86_64 13/26 2021-05-12T11:50:17.853 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 14/26 2021-05-12T11:50:18.088 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 15/26 2021-05-12T11:50:18.274 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:17.0.0-4087.g36c2cf2f.el8.x86_6 16/26 2021-05-12T11:50:18.352 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:17.0.0-4087.g36c2cf2f.el8.x86_6 16/26 2021-05-12T11:50:18.554 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/26 2021-05-12T11:50:18.811 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 18/26 2021-05-12T11:50:19.022 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2021-05-12T11:50:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x8 20/26 2021-05-12T11:50:23.088 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 21/26 2021-05-12T11:50:23.311 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 21/26 2021-05-12T11:50:24.019 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 21/26 2021-05-12T11:50:24.284 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 22/26 2021-05-12T11:50:24.389 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 22/26 2021-05-12T11:50:24.518 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:17.0.0-4087.g36c2cf2f.el8.x86_64 23/26 2021-05-12T11:50:41.404 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:17.0.0-4087.g36c2cf2f.el8.x86_64 23/26 2021-05-12T11:50:41.405 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2021-05-12T11:50:41.405 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2021-05-12T11:50:41.405 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2021-05-12T11:50:41.405 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /run 2021-05-12T11:50:41.405 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2021-05-12T11:50:41.406 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2021-05-12T11:50:41.406 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2021-05-12T11:50:41.406 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2021-05-12T11:50:41.406 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2021-05-12T11:50:41.407 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:50:42.681 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 24/26 2021-05-12T11:50:42.769 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 24/26 2021-05-12T11:50:42.769 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-12T11:50:43.017 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-12T11:50:43.017 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-12T11:50:43.800 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-12T11:50:43.801 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/26 2021-05-12T11:50:43.801 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2/26 2021-05-12T11:50:43.801 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 3/26 2021-05-12T11:50:43.802 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:17.0.0-4087.g36c2cf2f.el8.x86_64 4/26 2021-05-12T11:50:43.802 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 5/26 2021-05-12T11:50:43.802 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:17.0.0-4087.g36c2cf2f.el8.x86_6 6/26 2021-05-12T11:50:43.802 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/26 2021-05-12T11:50:43.802 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:17.0.0-4087.g36c2cf2f.el8. 8/26 2021-05-12T11:50:43.803 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x8 9/26 2021-05-12T11:50:43.803 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:17.0.0-4087.g36c2cf2f.el8.x86_64 10/26 2021-05-12T11:50:43.803 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:17.0.0-4087.g36c2cf2f.el8.x86_64 11/26 2021-05-12T11:50:43.803 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 12/26 2021-05-12T11:50:43.803 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 13/26 2021-05-12T11:50:43.804 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 14/26 2021-05-12T11:50:43.804 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 15/26 2021-05-12T11:50:43.804 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : leveldb-1.22-1.el8.x86_64 16/26 2021-05-12T11:50:43.804 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 17/26 2021-05-12T11:50:43.804 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 18/26 2021-05-12T11:50:43.805 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 19/26 2021-05-12T11:50:43.805 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/26 2021-05-12T11:50:43.805 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 21/26 2021-05-12T11:50:43.805 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 22/26 2021-05-12T11:50:43.806 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 23/26 2021-05-12T11:50:43.806 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-05-12T11:50:43.806 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 25/26 2021-05-12T11:50:44.879 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-05-12T11:50:44.879 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:50:53.943 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:50:49 AM UTC. 2021-05-12T11:50:55.892 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:50:55.893 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2021-05-12T11:50:55.893 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.893 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.893 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:50:55.893 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:50:55.894 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.894 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.894 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.894 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.895 INFO:teuthology.orchestra.run.smithi191.stdout: fmt-6.2.1-1.el8.x86_64 2021-05-12T11:50:55.895 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-05-12T11:50:55.895 INFO:teuthology.orchestra.run.smithi191.stdout: leveldb-1.22-1.el8.x86_64 2021-05-12T11:50:55.896 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.896 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.2-3.el8.x86_64 2021-05-12T11:50:55.896 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-05-12T11:50:55.896 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.896 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-05-12T11:50:55.897 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.897 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.3.1-3.el8.x86_64 2021-05-12T11:50:55.897 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-05-12T11:50:55.897 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.898 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.898 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.898 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-05-12T11:50:55.898 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.899 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.899 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:50:55.899 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:50:55.899 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:50:55.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test 2021-05-12T11:50:56.268 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:50:56.709 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-05-12T11:51:00.623 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-05-12T11:51:04.811 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:51:06.902 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:51:06.903 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2021-05-12T11:51:06.903 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2021-05-12T11:51:06.904 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2021-05-12T11:51:06.904 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:51:06.904 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 43 M 2021-05-12T11:51:06.904 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2021-05-12T11:51:06.905 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-05-12T11:51:06.905 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 159 k 2021-05-12T11:51:06.905 INFO:teuthology.orchestra.run.smithi191.stdout: libxslt x86_64 1.1.32-5.el8 rhel-8-for-x86_64-baseos-rpms 250 k 2021-05-12T11:51:06.905 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-05-12T11:51:06.905 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-05-12T11:51:06.906 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-05-12T11:51:06.906 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:51:06.906 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:51:06.906 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2021-05-12T11:51:06.907 INFO:teuthology.orchestra.run.smithi191.stdout:Install 7 Packages 2021-05-12T11:51:06.907 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:51:06.908 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 44 M 2021-05-12T11:51:06.908 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 243 M 2021-05-12T11:51:06.908 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:51:07.050 INFO:teuthology.orchestra.run.smithi191.stdout:(1/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 489 kB/s | 69 kB 00:00 2021-05-12T11:51:07.079 INFO:teuthology.orchestra.run.smithi191.stdout:(2/7): jq-1.5-12.el8.x86_64.rpm 5.4 MB/s | 161 kB 00:00 2021-05-12T11:51:07.099 INFO:teuthology.orchestra.run.smithi191.stdout:(3/7): oniguruma-6.8.2-2.el8.x86_64.rpm 9.3 MB/s | 187 kB 00:00 2021-05-12T11:51:07.112 INFO:teuthology.orchestra.run.smithi191.stdout:(4/7): libcephsqlite-17.0.0-4087.g36c2cf2f.el8. 780 kB/s | 159 kB 00:00 2021-05-12T11:51:07.122 INFO:teuthology.orchestra.run.smithi191.stdout:(5/7): socat-1.7.3.3-2.el8.x86_64.rpm 13 MB/s | 302 kB 00:00 2021-05-12T11:51:07.143 INFO:teuthology.orchestra.run.smithi191.stdout:(6/7): libxslt-1.1.32-5.el8.x86_64.rpm 8.3 MB/s | 250 kB 00:00 2021-05-12T11:51:09.508 INFO:teuthology.orchestra.run.smithi191.stdout:(7/7): ceph-test-17.0.0-4087.g36c2cf2f.el8.x86_ 16 MB/s | 43 MB 00:02 2021-05-12T11:51:09.511 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:51:09.511 INFO:teuthology.orchestra.run.smithi191.stdout:Total 17 MB/s | 44 MB 00:02 2021-05-12T11:51:09.515 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:51:09.539 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:51:09.539 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:51:09.790 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:51:09.793 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:51:10.306 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:51:10.587 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/7 2021-05-12T11:51:10.799 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-05-12T11:51:11.012 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-05-12T11:51:11.251 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-12T11:51:11.336 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-12T11:51:11.567 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-05-12T11:51:11.710 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:17.0.0-4087.g36c2cf2f.el8.x86_64 6/7 2021-05-12T11:51:16.288 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:17.0.0-4087.g36c2cf2f.el8.x86_64 6/7 2021-05-12T11:51:16.499 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/7 2021-05-12T11:51:16.743 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/7 2021-05-12T11:51:16.744 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/7 2021-05-12T11:51:16.744 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2/7 2021-05-12T11:51:16.744 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 3/7 2021-05-12T11:51:16.745 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.5-12.el8.x86_64 4/7 2021-05-12T11:51:16.745 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/7 2021-05-12T11:51:16.745 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/7 2021-05-12T11:51:17.330 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 7/7 2021-05-12T11:51:17.330 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:51:25.813 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:50:49 AM UTC. 2021-05-12T11:51:27.714 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:51:27.715 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:51:27.715 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:17.0.0-4087.g36c2cf2f.el8.x86_64 jq-1.5-12.el8.x86_64 2021-05-12T11:51:27.715 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:17.0.0-4087.g36c2cf2f.el8.x86_64 libxslt-1.1.32-5.el8.x86_64 2021-05-12T11:51:27.715 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-05-12T11:51:27.716 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-05-12T11:51:27.716 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:51:27.716 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:51:27.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph 2021-05-12T11:51:28.069 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:51:28.518 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 100 kB/s | 2.8 kB 00:00 2021-05-12T11:51:32.564 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-05-12T11:51:36.802 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:51:38.907 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:51:38.910 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================================= 2021-05-12T11:51:38.910 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2021-05-12T11:51:38.910 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================================= 2021-05-12T11:51:38.910 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:51:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 6.4 k 2021-05-12T11:51:38.911 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2021-05-12T11:51:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 2.2 M 2021-05-12T11:51:38.911 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 1.5 M 2021-05-12T11:51:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 204 k 2021-05-12T11:51:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.8 M 2021-05-12T11:51:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 16 M 2021-05-12T11:51:38.912 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asn1crypto noarch 0.24.0-3.el8 rhel-8-for-x86_64-baseos-rpms 181 k 2021-05-12T11:51:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-05-12T11:51:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-05-12T11:51:38.913 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-05-12T11:51:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-05-12T11:51:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-05-12T11:51:38.914 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 2.3-3.el8 rhel-8-for-x86_64-baseos-rpms 511 k 2021-05-12T11:51:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-05-12T11:51:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-05-12T11:51:38.915 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-05-12T11:51:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: python3-lxml x86_64 4.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2021-05-12T11:51:38.916 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-05-12T11:51:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-05-12T11:51:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-05-12T11:51:38.917 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-05-12T11:51:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-05-12T11:51:38.918 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-05-12T11:51:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-05-12T11:51:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-05-12T11:51:38.919 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-05-12T11:51:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-05-12T11:51:38.920 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-05-12T11:51:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-05-12T11:51:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-05-12T11:51:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webencodings noarch 0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-05-12T11:51:38.922 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-05-12T11:51:38.922 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-05-12T11:51:38.922 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-05-12T11:51:38.923 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-05-12T11:51:38.923 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2021-05-12T11:51:38.923 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-05-12T11:51:38.923 INFO:teuthology.orchestra.run.smithi191.stdout: python3-html5lib noarch 1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2021-05-12T11:51:38.924 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:51:38.924 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:51:38.924 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================================= 2021-05-12T11:51:38.924 INFO:teuthology.orchestra.run.smithi191.stdout:Install 37 Packages 2021-05-12T11:51:38.925 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:51:38.925 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 29 M 2021-05-12T11:51:38.926 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 123 M 2021-05-12T11:51:38.926 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:51:39.014 INFO:teuthology.orchestra.run.smithi191.stdout:(1/37): ceph-17.0.0-4087.g36c2cf2f.el8.x86_64.r 65 kB/s | 6.4 kB 00:00 2021-05-12T11:51:39.221 INFO:teuthology.orchestra.run.smithi191.stdout:(2/37): ceph-mgr-17.0.0-4087.g36c2cf2f.el8.x86_ 4.8 MB/s | 1.5 MB 00:00 2021-05-12T11:51:39.268 INFO:teuthology.orchestra.run.smithi191.stdout:(3/37): ceph-mds-17.0.0-4087.g36c2cf2f.el8.x86_ 6.3 MB/s | 2.2 MB 00:00 2021-05-12T11:51:39.329 INFO:teuthology.orchestra.run.smithi191.stdout:(4/37): ceph-mgr-modules-core-17.0.0-4087.g36c2 3.3 MB/s | 204 kB 00:00 2021-05-12T11:51:39.442 INFO:teuthology.orchestra.run.smithi191.stdout:(5/37): ceph-mon-17.0.0-4087.g36c2cf2f.el8.x86_ 8.9 MB/s | 3.8 MB 00:00 2021-05-12T11:51:39.471 INFO:teuthology.orchestra.run.smithi191.stdout:(6/37): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 307 kB/s | 44 kB 00:00 2021-05-12T11:51:39.552 INFO:teuthology.orchestra.run.smithi191.stdout:(7/37): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.1 MB/s | 173 kB 00:00 2021-05-12T11:51:39.584 INFO:teuthology.orchestra.run.smithi191.stdout:(8/37): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.3 MB/s | 185 kB 00:00 2021-05-12T11:51:39.607 INFO:teuthology.orchestra.run.smithi191.stdout:(9/37): python3-cssselect-0.9.2-10.el8.noarch.r 1.7 MB/s | 40 kB 00:00 2021-05-12T11:51:39.608 INFO:teuthology.orchestra.run.smithi191.stdout:(10/37): python3-cherrypy-18.4.0-1.el8.noarch.r 6.8 MB/s | 384 kB 00:00 2021-05-12T11:51:39.629 INFO:teuthology.orchestra.run.smithi191.stdout:(11/37): python3-jaraco-6.2-6.el8.noarch.rpm 504 kB/s | 11 kB 00:00 2021-05-12T11:51:39.635 INFO:teuthology.orchestra.run.smithi191.stdout:(12/37): python3-jaraco-functools-2.0-4.el8.noa 657 kB/s | 18 kB 00:00 2021-05-12T11:51:39.651 INFO:teuthology.orchestra.run.smithi191.stdout:(13/37): python3-logutils-0.3.5-11.el8.noarch.r 2.1 MB/s | 49 kB 00:00 2021-05-12T11:51:39.662 INFO:teuthology.orchestra.run.smithi191.stdout:(14/37): python3-more-itertools-7.2.0-3.el8.noa 2.1 MB/s | 59 kB 00:00 2021-05-12T11:51:39.689 INFO:teuthology.orchestra.run.smithi191.stdout:(15/37): python3-portend-2.6-1.el8.noarch.rpm 579 kB/s | 16 kB 00:00 2021-05-12T11:51:39.696 INFO:teuthology.orchestra.run.smithi191.stdout:(16/37): python3-pecan-1.3.2-9.el8.noarch.rpm 6.2 MB/s | 283 kB 00:00 2021-05-12T11:51:39.716 INFO:teuthology.orchestra.run.smithi191.stdout:(17/37): python3-simplegeneric-0.8.1-17.el8.noa 699 kB/s | 19 kB 00:00 2021-05-12T11:51:39.719 INFO:teuthology.orchestra.run.smithi191.stdout:(18/37): python3-singledispatch-3.4.0.3-18.el8. 1.1 MB/s | 24 kB 00:00 2021-05-12T11:51:39.742 INFO:teuthology.orchestra.run.smithi191.stdout:(19/37): python3-trustme-0.6.0-4.el8.noarch.rpm 1.2 MB/s | 27 kB 00:00 2021-05-12T11:51:39.744 INFO:teuthology.orchestra.run.smithi191.stdout:(20/37): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-05-12T11:51:39.774 INFO:teuthology.orchestra.run.smithi191.stdout:(21/37): python3-webob-1.8.5-1.el8.1.noarch.rpm 8.2 MB/s | 251 kB 00:00 2021-05-12T11:51:39.785 INFO:teuthology.orchestra.run.smithi191.stdout:(22/37): python3-waitress-1.2.1-2.el8.1.noarch. 5.3 MB/s | 241 kB 00:00 2021-05-12T11:51:39.801 INFO:teuthology.orchestra.run.smithi191.stdout:(23/37): python3-webtest-2.0.33-1.el8.noarch.rp 3.0 MB/s | 85 kB 00:00 2021-05-12T11:51:39.811 INFO:teuthology.orchestra.run.smithi191.stdout:(24/37): python3-zc-lockfile-2.0-2.el8.noarch.r 911 kB/s | 23 kB 00:00 2021-05-12T11:51:39.842 INFO:teuthology.orchestra.run.smithi191.stdout:(25/37): python3-html5lib-0.999999999-6.el8.noa 5.2 MB/s | 214 kB 00:00 2021-05-12T11:51:39.850 INFO:teuthology.orchestra.run.smithi191.stdout:(26/37): python3-mako-1.0.6-13.el8.noarch.rpm 4.0 MB/s | 157 kB 00:00 2021-05-12T11:51:39.869 INFO:teuthology.orchestra.run.smithi191.stdout:(27/37): python3-werkzeug-0.12.2-4.el8.noarch.r 17 MB/s | 457 kB 00:00 2021-05-12T11:51:39.876 INFO:teuthology.orchestra.run.smithi191.stdout:(28/37): python3-markupsafe-0.23-19.el8.x86_64. 1.5 MB/s | 39 kB 00:00 2021-05-12T11:51:39.893 INFO:teuthology.orchestra.run.smithi191.stdout:(29/37): python3-pytz-2017.2-9.el8.noarch.rpm 2.2 MB/s | 54 kB 00:00 2021-05-12T11:51:39.900 INFO:teuthology.orchestra.run.smithi191.stdout:(30/37): python3-webencodings-0.5.1-6.el8.noarc 1.1 MB/s | 27 kB 00:00 2021-05-12T11:51:40.044 INFO:teuthology.orchestra.run.smithi191.stdout:(31/37): ceph-osd-17.0.0-4087.g36c2cf2f.el8.x86 20 MB/s | 16 MB 00:00 2021-05-12T11:51:40.050 INFO:teuthology.orchestra.run.smithi191.stdout:(32/37): python3-pyOpenSSL-18.0.0-1.el8.noarch. 658 kB/s | 103 kB 00:00 2021-05-12T11:51:40.083 INFO:teuthology.orchestra.run.smithi191.stdout:(33/37): python3-lxml-4.2.3-1.el8.x86_64.rpm 8.2 MB/s | 1.5 MB 00:00 2021-05-12T11:51:40.084 INFO:teuthology.orchestra.run.smithi191.stdout:(34/37): python3-asn1crypto-0.24.0-3.el8.noarch 5.2 MB/s | 181 kB 00:00 2021-05-12T11:51:40.084 INFO:teuthology.orchestra.run.smithi191.stdout:(35/37): python3-pycparser-2.14-14.el8.noarch.r 2.9 MB/s | 109 kB 00:00 2021-05-12T11:51:40.107 INFO:teuthology.orchestra.run.smithi191.stdout:(36/37): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.8 MB/s | 238 kB 00:00 2021-05-12T11:51:40.115 INFO:teuthology.orchestra.run.smithi191.stdout:(37/37): python3-cryptography-2.3-3.el8.x86_64. 16 MB/s | 511 kB 00:00 2021-05-12T11:51:40.118 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:51:40.118 INFO:teuthology.orchestra.run.smithi191.stdout:Total 24 MB/s | 29 MB 00:01 2021-05-12T11:51:40.129 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:51:40.168 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:51:40.168 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:51:40.416 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:51:40.420 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:51:41.735 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:51:41.981 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/37 2021-05-12T11:51:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/37 2021-05-12T11:51:42.725 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 3/37 2021-05-12T11:51:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 4/37 2021-05-12T11:51:43.327 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 5/37 2021-05-12T11:51:43.592 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 6/37 2021-05-12T11:51:43.767 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 7/37 2021-05-12T11:51:43.981 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 8/37 2021-05-12T11:51:44.205 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 9/37 2021-05-12T11:51:44.471 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 10/37 2021-05-12T11:51:44.694 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 11/37 2021-05-12T11:51:44.926 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 12/37 2021-05-12T11:51:45.131 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 13/37 2021-05-12T11:51:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 14/37 2021-05-12T11:51:45.653 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 15/37 2021-05-12T11:51:45.930 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 16/37 2021-05-12T11:51:46.169 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 17/37 2021-05-12T11:51:46.324 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 18/37 2021-05-12T11:51:46.531 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 19/37 2021-05-12T11:51:46.681 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 20/37 2021-05-12T11:51:46.851 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 21/37 2021-05-12T11:51:47.049 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 22/37 2021-05-12T11:51:49.070 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/37 2021-05-12T11:51:49.585 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 24/37 2021-05-12T11:51:49.828 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-2.6-1.el8.noarch 25/37 2021-05-12T11:51:50.082 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 26/37 2021-05-12T11:51:50.441 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 27/37 2021-05-12T11:51:50.741 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/37 2021-05-12T11:51:50.979 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 29/37 2021-05-12T11:51:51.305 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 30/37 2021-05-12T11:51:51.637 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 31/37 2021-05-12T11:51:52.013 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-4087.g36c2cf2f.el8. 32/37 2021-05-12T11:51:52.202 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 33/37 2021-05-12T11:51:53.703 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 33/37 2021-05-12T11:51:53.876 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 34/37 2021-05-12T11:51:54.280 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 34/37 2021-05-12T11:51:54.434 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:17.0.0-4087.g36c2cf2f.el8.x86_64 35/37 2021-05-12T11:51:54.655 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:17.0.0-4087.g36c2cf2f.el8.x86_64 35/37 2021-05-12T11:51:55.074 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:17.0.0-4087.g36c2cf2f.el8.x86_64 36/37 2021-05-12T11:51:55.095 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:17.0.0-4087.g36c2cf2f.el8.x86_64 36/37 2021-05-12T11:51:55.192 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:17.0.0-4087.g36c2cf2f.el8.x86_64 37/37 2021-05-12T11:51:55.584 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-2:17.0.0-4087.g36c2cf2f.el8.x86_64 37/37 2021-05-12T11:51:55.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/37 2021-05-12T11:51:55.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2/37 2021-05-12T11:51:55.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 3/37 2021-05-12T11:51:55.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:17.0.0-4087.g36c2cf2f.el8.x86_64 4/37 2021-05-12T11:51:55.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 5/37 2021-05-12T11:51:55.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-4087.g36c2cf2f.el8. 6/37 2021-05-12T11:51:55.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/37 2021-05-12T11:51:55.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/37 2021-05-12T11:51:55.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/37 2021-05-12T11:51:55.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/37 2021-05-12T11:51:55.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 11/37 2021-05-12T11:51:55.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 12/37 2021-05-12T11:51:55.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 13/37 2021-05-12T11:51:55.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 14/37 2021-05-12T11:51:55.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 15/37 2021-05-12T11:51:55.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 16/37 2021-05-12T11:51:55.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-2.6-1.el8.noarch 17/37 2021-05-12T11:51:55.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 18/37 2021-05-12T11:51:55.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 19/37 2021-05-12T11:51:55.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 20/37 2021-05-12T11:51:55.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 21/37 2021-05-12T11:51:55.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 22/37 2021-05-12T11:51:55.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 23/37 2021-05-12T11:51:55.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 24/37 2021-05-12T11:51:55.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 25/37 2021-05-12T11:51:55.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 26/37 2021-05-12T11:51:55.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/37 2021-05-12T11:51:55.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/37 2021-05-12T11:51:55.592 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/37 2021-05-12T11:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 30/37 2021-05-12T11:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 31/37 2021-05-12T11:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 32/37 2021-05-12T11:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 33/37 2021-05-12T11:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 34/37 2021-05-12T11:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 35/37 2021-05-12T11:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 36/37 2021-05-12T11:51:56.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 37/37 2021-05-12T11:51:56.585 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:52:05.294 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:51:21 AM UTC. 2021-05-12T11:52:07.223 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:07.223 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:52:07.224 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:52:07.224 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:52:07.224 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:52:07.225 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:52:07.225 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:52:07.225 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:52:07.226 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-05-12T11:52:07.226 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-05-12T11:52:07.226 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-05-12T11:52:07.227 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-05-12T11:52:07.227 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-05-12T11:52:07.227 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-05-12T11:52:07.228 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-05-12T11:52:07.228 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-05-12T11:52:07.229 INFO:teuthology.orchestra.run.smithi191.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-05-12T11:52:07.229 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-6.2-6.el8.noarch 2021-05-12T11:52:07.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-05-12T11:52:07.230 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-05-12T11:52:07.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-05-12T11:52:07.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.0.6-13.el8.noarch 2021-05-12T11:52:07.231 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-05-12T11:52:07.232 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-05-12T11:52:07.232 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-05-12T11:52:07.233 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-2.6-1.el8.noarch 2021-05-12T11:52:07.233 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-05-12T11:52:07.233 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.14-14.el8.noarch 2021-05-12T11:52:07.234 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2017.2-9.el8.noarch 2021-05-12T11:52:07.234 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-05-12T11:52:07.235 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-05-12T11:52:07.235 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-05-12T11:52:07.235 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-05-12T11:52:07.235 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-05-12T11:52:07.236 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-05-12T11:52:07.236 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-05-12T11:52:07.236 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-05-12T11:52:07.237 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-05-12T11:52:07.237 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-05-12T11:52:07.237 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:07.237 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:52:07.331 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base 2021-05-12T11:52:07.611 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:52:08.063 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 102 kB/s | 2.8 kB 00:00 2021-05-12T11:52:12.109 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-05-12T11:52:16.458 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:52:18.243 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:52:18.598 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:52:18.599 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:52:18.599 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:52:18.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install cephadm 2021-05-12T11:52:18.955 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:52:19.431 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-05-12T11:52:19.578 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-05-12T11:52:21.102 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:52:21.103 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:52:21.103 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2021-05-12T11:52:21.104 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:52:21.104 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:52:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 67 k 2021-05-12T11:52:21.104 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:21.104 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:52:21.105 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:52:21.105 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2021-05-12T11:52:21.105 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:21.105 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 67 k 2021-05-12T11:52:21.106 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 296 k 2021-05-12T11:52:21.106 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:52:21.248 INFO:teuthology.orchestra.run.smithi191.stdout:cephadm-17.0.0-4087.g36c2cf2f.el8.noarch.rpm 464 kB/s | 67 kB 00:00 2021-05-12T11:52:21.249 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:52:21.249 INFO:teuthology.orchestra.run.smithi191.stdout:Total 460 kB/s | 67 kB 00:00 2021-05-12T11:52:21.249 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:52:21.256 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:52:21.256 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:52:21.258 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:52:21.262 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:52:21.333 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:52:23.384 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/1 2021-05-12T11:52:23.633 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/1 2021-05-12T11:52:23.802 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/1 2021-05-12T11:52:24.242 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/1 2021-05-12T11:52:24.242 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:52:32.662 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:52:01 AM UTC. 2021-05-12T11:52:34.559 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:34.559 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:52:34.559 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:52:34.560 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:34.560 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:52:34.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache 2021-05-12T11:52:34.959 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:52:35.485 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-05-12T11:52:39.487 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-05-12T11:52:43.693 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:52:45.843 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:52:45.845 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:52:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2021-05-12T11:52:45.845 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:52:45.845 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:52:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 167 k 2021-05-12T11:52:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:45.846 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:52:45.846 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:52:45.847 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2021-05-12T11:52:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:45.847 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 167 k 2021-05-12T11:52:45.847 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 633 k 2021-05-12T11:52:45.848 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:52:46.014 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-17.0.0-4087.g36c2cf 992 kB/s | 167 kB 00:00 2021-05-12T11:52:46.014 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:52:46.015 INFO:teuthology.orchestra.run.smithi191.stdout:Total 984 kB/s | 167 kB 00:00 2021-05-12T11:52:46.015 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:52:46.027 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:52:46.027 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:52:46.078 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:52:46.082 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:52:46.208 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:52:46.376 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-4087.g36c2cf2f. 1/1 2021-05-12T11:52:46.735 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-4087.g36c2cf2f. 1/1 2021-05-12T11:52:47.206 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-4087.g36c2cf2f. 1/1 2021-05-12T11:52:47.206 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:52:55.868 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:52:28 AM UTC. 2021-05-12T11:52:57.844 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:57.844 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:52:57.844 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:52:57.845 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:52:57.845 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:52:58.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr 2021-05-12T11:52:58.412 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:52:58.882 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-05-12T11:53:02.840 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-05-12T11:53:06.982 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:53:08.796 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:53:09.138 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:53:09.139 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:53:09.139 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:53:09.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-dashboard 2021-05-12T11:53:09.491 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:53:09.973 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 87 kB/s | 2.8 kB 00:00 2021-05-12T11:53:10.119 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-05-12T11:53:11.637 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:53:11.638 INFO:teuthology.orchestra.run.smithi191.stdout:=================================================================================================== 2021-05-12T11:53:11.638 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2021-05-12T11:53:11.639 INFO:teuthology.orchestra.run.smithi191.stdout:=================================================================================================== 2021-05-12T11:53:11.639 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:53:11.639 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 2.3 M 2021-05-12T11:53:11.639 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2021-05-12T11:53:11.639 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 24 k 2021-05-12T11:53:11.640 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 8.9 k 2021-05-12T11:53:11.640 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-05-12T11:53:11.640 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-05-12T11:53:11.640 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-05-12T11:53:11.640 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:53:11.641 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:53:11.641 INFO:teuthology.orchestra.run.smithi191.stdout:=================================================================================================== 2021-05-12T11:53:11.641 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2021-05-12T11:53:11.641 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:53:11.642 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.6 M 2021-05-12T11:53:11.642 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 15 M 2021-05-12T11:53:11.643 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:53:11.749 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): ceph-prometheus-alerts-17.0.0-4087.g36c2 82 kB/s | 8.9 kB 00:00 2021-05-12T11:53:11.762 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): ceph-grafana-dashboards-17.0.0-4087.g36c 197 kB/s | 24 kB 00:00 2021-05-12T11:53:11.838 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 377 kB/s | 34 kB 00:00 2021-05-12T11:53:11.868 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-05-12T11:53:11.938 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): ceph-mgr-dashboard-17.0.0-4087.g36c2cf2f 7.6 MB/s | 2.3 MB 00:00 2021-05-12T11:53:11.963 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 978 kB/s | 196 kB 00:00 2021-05-12T11:53:11.965 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:53:11.966 INFO:teuthology.orchestra.run.smithi191.stdout:Total 7.8 MB/s | 2.6 MB 00:00 2021-05-12T11:53:11.967 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:53:11.976 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:53:11.976 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:53:12.052 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:53:12.056 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:53:12.311 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:53:12.474 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-05-12T11:53:12.715 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-05-12T11:53:12.907 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-05-12T11:53:13.037 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-4087.g36c2cf2f.el8.n 4/6 2021-05-12T11:53:14.114 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-4087.g36c2cf2f.el8. 5/6 2021-05-12T11:53:14.386 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-4087.g36c2cf2f.el8.noarc 6/6 2021-05-12T11:53:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-4087.g36c2cf2f.el8.noarc 6/6 2021-05-12T11:53:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-4087.g36c2cf2f.el8. 1/6 2021-05-12T11:53:14.624 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-4087.g36c2cf2f.el8.noarc 2/6 2021-05-12T11:53:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-4087.g36c2cf2f.el8.n 3/6 2021-05-12T11:53:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-05-12T11:53:14.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-05-12T11:53:15.067 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-05-12T11:53:15.067 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:53:23.509 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:52:51 AM UTC. 2021-05-12T11:53:25.431 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:53:25.431 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:53:25.432 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:53:25.432 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:53:25.432 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:53:25.432 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-05-12T11:53:25.432 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-05-12T11:53:25.433 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.4.1-12.el8.noarch 2021-05-12T11:53:25.433 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:53:25.433 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:53:25.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-diskprediction-local 2021-05-12T11:53:25.840 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:53:26.357 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-05-12T11:53:30.275 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-05-12T11:53:34.427 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:53:36.540 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:53:36.541 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================================================== 2021-05-12T11:53:36.542 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2021-05-12T11:53:36.542 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================================================== 2021-05-12T11:53:36.542 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:53:36.542 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 7.4 M 2021-05-12T11:53:36.543 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2021-05-12T11:53:36.543 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-05-12T11:53:36.543 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-05-12T11:53:36.543 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2021-05-12T11:53:36.543 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2021-05-12T11:53:36.544 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-05-12T11:53:36.544 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-05-12T11:53:36.544 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-05-12T11:53:36.544 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:53:36.545 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:53:36.545 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================================================== 2021-05-12T11:53:36.545 INFO:teuthology.orchestra.run.smithi191.stdout:Install 8 Packages 2021-05-12T11:53:36.545 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:53:36.546 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 35 M 2021-05-12T11:53:36.546 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 196 M 2021-05-12T11:53:36.547 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:53:36.612 INFO:teuthology.orchestra.run.smithi191.stdout:(1/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 54 MB/s | 3.7 MB 00:00 2021-05-12T11:53:36.637 INFO:teuthology.orchestra.run.smithi191.stdout:(2/8): openblas-0.3.3-5.el8.x86_64.rpm 46 MB/s | 4.3 MB 00:00 2021-05-12T11:53:36.639 INFO:teuthology.orchestra.run.smithi191.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 8.2 MB/s | 225 kB 00:00 2021-05-12T11:53:36.691 INFO:teuthology.orchestra.run.smithi191.stdout:(4/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 82 MB/s | 4.4 MB 00:00 2021-05-12T11:53:36.714 INFO:teuthology.orchestra.run.smithi191.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 27 MB/s | 639 kB 00:00 2021-05-12T11:53:36.735 INFO:teuthology.orchestra.run.smithi191.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 8.4 MB/s | 169 kB 00:00 2021-05-12T11:53:36.857 INFO:teuthology.orchestra.run.smithi191.stdout:(7/8): python3-scipy-1.0.0-20.module+el8.1.0+33 67 MB/s | 15 MB 00:00 2021-05-12T11:53:37.102 INFO:teuthology.orchestra.run.smithi191.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-408 13 MB/s | 7.4 MB 00:00 2021-05-12T11:53:37.103 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:53:37.103 INFO:teuthology.orchestra.run.smithi191.stdout:Total 63 MB/s | 35 MB 00:00 2021-05-12T11:53:37.112 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:53:37.131 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:53:37.131 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:53:37.443 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:53:37.447 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:53:37.991 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:53:38.167 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-12T11:53:38.328 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-12T11:53:38.475 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-12T11:53:38.996 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-12T11:53:39.141 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-05-12T11:53:39.712 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-05-12T11:53:39.891 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-12T11:53:40.604 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-12T11:53:40.982 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-05-12T11:53:41.241 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-12T11:53:43.247 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-12T11:53:44.544 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-05-12T11:53:44.732 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-4087.g36c2cf2 8/8 2021-05-12T11:53:45.026 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-4087.g36c2cf2 8/8 2021-05-12T11:53:45.026 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-4087.g36c2cf2 1/8 2021-05-12T11:53:45.027 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 2/8 2021-05-12T11:53:45.027 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 3/8 2021-05-12T11:53:45.027 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 4/8 2021-05-12T11:53:45.027 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 5/8 2021-05-12T11:53:45.027 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 6/8 2021-05-12T11:53:45.028 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-05-12T11:53:45.518 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-05-12T11:53:45.518 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:53:54.018 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:53:19 AM UTC. 2021-05-12T11:53:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:53:55.965 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:53:55.965 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:53:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-05-12T11:53:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-05-12T11:53:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.3-5.el8.x86_64 2021-05-12T11:53:55.966 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-05-12T11:53:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-05-12T11:53:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-05-12T11:53:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-05-12T11:53:55.967 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:53:55.967 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:53:56.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-rook 2021-05-12T11:53:56.330 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:53:56.810 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-05-12T11:54:00.789 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-05-12T11:54:05.051 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:54:07.183 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:54:07.184 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================================== 2021-05-12T11:54:07.184 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2021-05-12T11:54:07.184 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================================== 2021-05-12T11:54:07.185 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:54:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 26 k 2021-05-12T11:54:07.185 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2021-05-12T11:54:07.185 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-05-12T11:54:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-05-12T11:54:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-05-12T11:54:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-05-12T11:54:07.186 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-05-12T11:54:07.187 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-05-12T11:54:07.187 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-05-12T11:54:07.187 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-05-12T11:54:07.187 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-05-12T11:54:07.187 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-05-12T11:54:07.188 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-05-12T11:54:07.188 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-05-12T11:54:07.188 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:54:07.188 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:54:07.189 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================================== 2021-05-12T11:54:07.189 INFO:teuthology.orchestra.run.smithi191.stdout:Install 13 Packages 2021-05-12T11:54:07.189 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:54:07.189 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 1.9 M 2021-05-12T11:54:07.189 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 24 M 2021-05-12T11:54:07.190 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:54:07.312 INFO:teuthology.orchestra.run.smithi191.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 127 kB/s | 16 kB 00:00 2021-05-12T11:54:07.364 INFO:teuthology.orchestra.run.smithi191.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 184 kB/s | 33 kB 00:00 2021-05-12T11:54:07.378 INFO:teuthology.orchestra.run.smithi191.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.2 MB/s | 81 kB 00:00 2021-05-12T11:54:07.401 INFO:teuthology.orchestra.run.smithi191.stdout:(4/13): python3-rsa-3.4.2-16.el8.noarch.rpm 3.1 MB/s | 73 kB 00:00 2021-05-12T11:54:07.424 INFO:teuthology.orchestra.run.smithi191.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 2.6 MB/s | 61 kB 00:00 2021-05-12T11:54:07.448 INFO:teuthology.orchestra.run.smithi191.stdout:(6/13): ceph-mgr-rook-17.0.0-4087.g36c2cf2f.el8 99 kB/s | 26 kB 00:00 2021-05-12T11:54:07.456 INFO:teuthology.orchestra.run.smithi191.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 3.3 MB/s | 110 kB 00:00 2021-05-12T11:54:07.480 INFO:teuthology.orchestra.run.smithi191.stdout:(8/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 846 kB/s | 27 kB 00:00 2021-05-12T11:54:07.487 INFO:teuthology.orchestra.run.smithi191.stdout:(9/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.1 MB/s | 126 kB 00:00 2021-05-12T11:54:07.499 INFO:teuthology.orchestra.run.smithi191.stdout:(10/13): python3-jsonpointer-1.10-11.el8.noarch 1.0 MB/s | 20 kB 00:00 2021-05-12T11:54:07.511 INFO:teuthology.orchestra.run.smithi191.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 6.4 MB/s | 155 kB 00:00 2021-05-12T11:54:07.521 INFO:teuthology.orchestra.run.smithi191.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 1.9 MB/s | 43 kB 00:00 2021-05-12T11:54:07.554 INFO:teuthology.orchestra.run.smithi191.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.1 MB/s | 1.2 MB 00:00 2021-05-12T11:54:07.555 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:54:07.555 INFO:teuthology.orchestra.run.smithi191.stdout:Total 5.2 MB/s | 1.9 MB 00:00 2021-05-12T11:54:07.560 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:54:07.573 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:54:07.573 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:54:07.643 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:54:07.647 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:54:08.173 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:54:08.403 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-05-12T11:54:08.630 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-05-12T11:54:08.902 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-05-12T11:54:09.167 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2021-05-12T11:54:09.409 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2021-05-12T11:54:09.602 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2021-05-12T11:54:09.836 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2021-05-12T11:54:11.001 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-05-12T11:54:11.414 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-05-12T11:54:11.693 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-05-12T11:54:12.870 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-05-12T11:54:13.110 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-05-12T11:54:13.318 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:17.0.0-4087.g36c2cf2f.el8.noarch 13/13 2021-05-12T11:54:13.555 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-4087.g36c2cf2f.el8.noarch 13/13 2021-05-12T11:54:13.555 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/13 2021-05-12T11:54:13.555 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-05-12T11:54:13.556 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-05-12T11:54:13.556 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-05-12T11:54:13.556 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-05-12T11:54:13.556 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-05-12T11:54:13.556 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-05-12T11:54:13.557 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 8/13 2021-05-12T11:54:13.557 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 9/13 2021-05-12T11:54:13.557 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 10/13 2021-05-12T11:54:13.557 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-05-12T11:54:13.557 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2021-05-12T11:54:14.064 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2021-05-12T11:54:14.064 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:54:22.437 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:53:49 AM UTC. 2021-05-12T11:54:24.408 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:54:24.408 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:54:24.409 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:54:24.409 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-05-12T11:54:24.409 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-05-12T11:54:24.409 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-05-12T11:54:24.409 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-05-12T11:54:24.410 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-05-12T11:54:24.410 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-05-12T11:54:24.410 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-05-12T11:54:24.410 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-05-12T11:54:24.411 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-05-12T11:54:24.411 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-05-12T11:54:24.411 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-05-12T11:54:24.411 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-05-12T11:54:24.411 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:54:24.412 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:54:24.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-cephadm 2021-05-12T11:54:24.774 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:54:25.249 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-05-12T11:54:29.223 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-05-12T11:54:33.432 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:54:35.535 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:54:35.536 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================================== 2021-05-12T11:54:35.536 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2021-05-12T11:54:35.537 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================================== 2021-05-12T11:54:35.537 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:54:35.537 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 84 k 2021-05-12T11:54:35.537 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2021-05-12T11:54:35.538 INFO:teuthology.orchestra.run.smithi191.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-05-12T11:54:35.538 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-05-12T11:54:35.538 INFO:teuthology.orchestra.run.smithi191.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-05-12T11:54:35.538 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-05-12T11:54:35.538 INFO:teuthology.orchestra.run.smithi191.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-05-12T11:54:35.539 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:54:35.539 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:54:35.539 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================================== 2021-05-12T11:54:35.539 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2021-05-12T11:54:35.539 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:54:35.540 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 5.8 M 2021-05-12T11:54:35.540 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 24 M 2021-05-12T11:54:35.541 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:54:35.722 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): ceph-mgr-cephadm-17.0.0-4087.g36c2cf2f.e 455 kB/s | 84 kB 00:00 2021-05-12T11:54:35.749 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 100 kB/s | 21 kB 00:00 2021-05-12T11:54:35.807 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): python3-babel-2.5.1-5.el8.noarch.rpm 83 MB/s | 4.8 MB 00:00 2021-05-12T11:54:35.829 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 24 MB/s | 538 kB 00:00 2021-05-12T11:54:35.889 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): python3-remoto-1.1.4-4.el8.noarch.rpm 287 kB/s | 48 kB 00:00 2021-05-12T11:54:35.906 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 967 kB/s | 355 kB 00:00 2021-05-12T11:54:35.907 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:54:35.908 INFO:teuthology.orchestra.run.smithi191.stdout:Total 16 MB/s | 5.8 MB 00:00 2021-05-12T11:54:35.910 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:54:35.919 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:54:35.919 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:54:35.996 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:54:36.000 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:54:37.028 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:54:37.387 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2021-05-12T11:54:37.654 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2021-05-12T11:54:37.937 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2021-05-12T11:54:38.212 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2021-05-12T11:54:38.522 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 5/6 2021-05-12T11:54:38.697 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 6/6 2021-05-12T11:54:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 6/6 2021-05-12T11:54:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/6 2021-05-12T11:54:38.921 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2021-05-12T11:54:38.922 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2021-05-12T11:54:38.922 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 4/6 2021-05-12T11:54:38.922 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 5/6 2021-05-12T11:54:39.382 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 6/6 2021-05-12T11:54:39.383 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:54:47.902 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:54:18 AM UTC. 2021-05-12T11:54:49.843 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:54:49.844 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:54:49.844 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:54:49.844 INFO:teuthology.orchestra.run.smithi191.stdout: python3-apipkg-1.5-6.el8.noarch 2021-05-12T11:54:49.844 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.5.1-5.el8.noarch 2021-05-12T11:54:49.844 INFO:teuthology.orchestra.run.smithi191.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-05-12T11:54:49.845 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-05-12T11:54:49.845 INFO:teuthology.orchestra.run.smithi191.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-05-12T11:54:49.845 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:54:49.845 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:54:49.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse 2021-05-12T11:54:50.213 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:54:50.677 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-05-12T11:54:54.637 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-05-12T11:54:58.871 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:55:01.009 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:55:01.010 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2021-05-12T11:55:01.010 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2021-05-12T11:55:01.011 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2021-05-12T11:55:01.011 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:55:01.011 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 837 k 2021-05-12T11:55:01.011 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2021-05-12T11:55:01.012 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-05-12T11:55:01.012 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-common x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2021-05-12T11:55:01.012 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:01.012 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:55:01.012 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2021-05-12T11:55:01.013 INFO:teuthology.orchestra.run.smithi191.stdout:Install 3 Packages 2021-05-12T11:55:01.013 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:01.014 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 940 k 2021-05-12T11:55:01.014 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 3.3 M 2021-05-12T11:55:01.014 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:55:01.054 INFO:teuthology.orchestra.run.smithi191.stdout:(1/3): fuse-common-3.2.1-12.el8.x86_64.rpm 504 kB/s | 21 kB 00:00 2021-05-12T11:55:01.058 INFO:teuthology.orchestra.run.smithi191.stdout:(2/3): fuse-2.9.7-12.el8.x86_64.rpm 1.7 MB/s | 83 kB 00:00 2021-05-12T11:55:01.271 INFO:teuthology.orchestra.run.smithi191.stdout:(3/3): ceph-fuse-17.0.0-4087.g36c2cf2f.el8.x86_ 3.1 MB/s | 837 kB 00:00 2021-05-12T11:55:01.273 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:55:01.273 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.5 MB/s | 940 kB 00:00 2021-05-12T11:55:01.274 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:55:01.295 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:55:01.296 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:55:01.370 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:55:01.373 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:55:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:55:01.554 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2021-05-12T11:55:01.554 INFO:teuthology.orchestra.run.smithi191.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-05-12T11:55:01.555 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:02.060 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2021-05-12T11:55:02.252 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 3/3 2021-05-12T11:55:02.591 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 3/3 2021-05-12T11:55:02.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/3 2021-05-12T11:55:02.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 2/3 2021-05-12T11:55:03.109 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 3/3 2021-05-12T11:55:03.109 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:55:11.831 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:54:43 AM UTC. 2021-05-12T11:55:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:13.797 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:55:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-05-12T11:55:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-05-12T11:55:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:13.798 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:55:13.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel 2021-05-12T11:55:14.170 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:55:14.649 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-05-12T11:55:18.576 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-05-12T11:55:22.802 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:55:24.919 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:55:24.920 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:55:24.920 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2021-05-12T11:55:24.920 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:55:24.921 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:55:24.921 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 125 k 2021-05-12T11:55:24.921 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:24.921 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:55:24.921 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:55:24.922 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2021-05-12T11:55:24.922 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:24.922 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 125 k 2021-05-12T11:55:24.922 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 439 k 2021-05-12T11:55:24.923 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:55:25.090 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-17.0.0-4087.g36c2cf2f.el8.x86_64 739 kB/s | 125 kB 00:00 2021-05-12T11:55:25.091 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:55:25.091 INFO:teuthology.orchestra.run.smithi191.stdout:Total 733 kB/s | 125 kB 00:00 2021-05-12T11:55:25.091 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:55:25.097 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:55:25.098 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:55:25.154 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:55:25.158 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:55:25.275 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:55:25.461 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:55:25.700 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:55:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:55:26.181 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:55:34.648 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:55:07 AM UTC. 2021-05-12T11:55:36.599 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:36.599 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:55:36.600 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:55:36.600 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:36.600 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:55:36.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2 2021-05-12T11:55:36.973 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:55:37.448 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-05-12T11:55:41.394 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 87 kB/s | 2.4 kB 00:00 2021-05-12T11:55:45.580 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:55:47.355 INFO:teuthology.orchestra.run.smithi191.stdout:Package libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:55:47.697 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:55:47.698 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:55:47.698 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:55:47.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs-devel 2021-05-12T11:55:48.074 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:55:48.535 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 103 kB/s | 2.8 kB 00:00 2021-05-12T11:55:48.682 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 86 kB/s | 2.4 kB 00:00 2021-05-12T11:55:50.198 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:55:50.199 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:55:50.200 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2021-05-12T11:55:50.200 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:55:50.200 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:55:50.200 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 24 k 2021-05-12T11:55:50.201 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:50.201 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:55:50.201 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:55:50.201 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2021-05-12T11:55:50.202 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:55:50.202 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 24 k 2021-05-12T11:55:50.202 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 100 k 2021-05-12T11:55:50.203 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:55:50.321 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs-devel-17.0.0-4087.g36c2cf2f.el8.x86_6 195 kB/s | 24 kB 00:00 2021-05-12T11:55:50.322 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:55:50.323 INFO:teuthology.orchestra.run.smithi191.stdout:Total 193 kB/s | 24 kB 00:00 2021-05-12T11:55:50.323 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:55:50.326 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:55:50.327 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:55:50.328 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:55:50.332 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:55:50.391 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:55:50.566 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:55:50.757 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:55:51.222 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:55:51.223 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:55:59.749 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:55:30 AM UTC. 2021-05-12T11:56:02.244 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:56:02.245 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:56:02.245 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:02.245 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:56:02.245 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:56:02.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2 2021-05-12T11:56:02.623 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:56:03.107 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-05-12T11:56:07.143 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-05-12T11:56:11.431 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:56:13.184 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:56:13.525 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:56:13.525 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:56:13.526 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:56:13.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1 2021-05-12T11:56:13.908 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:56:14.370 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-05-12T11:56:14.515 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 88 kB/s | 2.4 kB 00:00 2021-05-12T11:56:15.691 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:56:16.030 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:56:16.031 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:56:16.031 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:56:16.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados 2021-05-12T11:56:16.415 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:56:16.876 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-05-12T11:56:17.021 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-05-12T11:56:18.191 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rados-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:56:18.528 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:56:18.529 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:56:18.529 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:56:18.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw 2021-05-12T11:56:19.363 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:56:19.864 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2021-05-12T11:56:20.011 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-05-12T11:56:21.198 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:56:21.545 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:56:21.546 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:56:21.546 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:56:21.703 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs 2021-05-12T11:56:21.993 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:56:22.490 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-05-12T11:56:22.637 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-05-12T11:56:23.848 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-cephfs-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:56:24.185 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:56:24.186 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:56:24.187 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:56:24.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd 2021-05-12T11:56:24.560 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:56:25.051 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-05-12T11:56:25.201 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-05-12T11:56:26.364 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:56:26.703 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:56:26.704 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2021-05-12T11:56:26.705 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:56:26.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse 2021-05-12T11:56:27.061 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:56:27.518 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-05-12T11:56:27.665 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-05-12T11:56:29.185 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:56:29.186 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:56:29.186 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2021-05-12T11:56:29.186 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:56:29.186 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:56:29.187 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 90 k 2021-05-12T11:56:29.187 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:56:29.187 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:56:29.187 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:56:29.188 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2021-05-12T11:56:29.188 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:56:29.188 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 90 k 2021-05-12T11:56:29.188 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 320 k 2021-05-12T11:56:29.189 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:56:29.331 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-17.0.0-4087.g36c2cf2f.el8.x86_64.rpm 623 kB/s | 90 kB 00:00 2021-05-12T11:56:29.332 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:56:29.332 INFO:teuthology.orchestra.run.smithi191.stdout:Total 617 kB/s | 90 kB 00:00 2021-05-12T11:56:29.333 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:56:29.343 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:56:29.344 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:56:29.399 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:56:29.403 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:56:29.520 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:56:29.691 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:56:29.927 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:56:30.417 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:56:30.417 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:56:38.976 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:55:55 AM UTC. 2021-05-12T11:56:40.883 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:56:40.883 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:56:40.884 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:40.884 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:56:40.884 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:56:40.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2021-05-12T11:56:41.254 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:56:41.742 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 100 kB/s | 2.8 kB 00:00 2021-05-12T11:56:45.795 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 86 kB/s | 2.4 kB 00:00 2021-05-12T11:56:50.264 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:56:52.373 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:56:52.374 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:56:52.374 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2021-05-12T11:56:52.374 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:56:52.375 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:56:52.375 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.7 M 2021-05-12T11:56:52.375 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:56:52.375 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:56:52.376 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:56:52.376 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2021-05-12T11:56:52.376 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:56:52.376 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.7 M 2021-05-12T11:56:52.376 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 21 M 2021-05-12T11:56:52.377 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:56:53.211 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-17.0.0-4087.g36c2cf2f.el8.x86_64.rpm 4.4 MB/s | 3.7 MB 00:00 2021-05-12T11:56:53.212 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:56:53.213 INFO:teuthology.orchestra.run.smithi191.stdout:Total 4.4 MB/s | 3.7 MB 00:00 2021-05-12T11:56:53.213 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:56:53.227 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:56:53.227 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:56:53.297 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:56:53.301 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:56:53.851 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:56:54.056 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:56:54.408 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:56:54.923 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:56:54.923 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:57:03.674 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:56:34 AM UTC. 2021-05-12T11:57:05.600 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:57:05.601 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:57:05.601 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:05.601 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:57:05.601 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:57:05.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd 2021-05-12T11:57:05.960 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2021-05-12T11:57:06.441 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-05-12T11:57:10.415 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-05-12T11:57:14.632 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:49:58 AM UTC. 2021-05-12T11:57:16.757 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2021-05-12T11:57:16.758 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:57:16.758 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2021-05-12T11:57:16.759 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:57:16.759 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2021-05-12T11:57:16.759 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 171 k 2021-05-12T11:57:16.760 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:57:16.760 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2021-05-12T11:57:16.760 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2021-05-12T11:57:16.760 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2021-05-12T11:57:16.761 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:57:16.761 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 171 k 2021-05-12T11:57:16.761 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 671 k 2021-05-12T11:57:16.761 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2021-05-12T11:57:16.929 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-17.0.0-4087.g36c2cf2f.el8.x86_64.rpm 1.0 MB/s | 171 kB 00:00 2021-05-12T11:57:16.930 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:16.930 INFO:teuthology.orchestra.run.smithi191.stdout:Total 995 kB/s | 171 kB 00:00 2021-05-12T11:57:16.930 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2021-05-12T11:57:16.942 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2021-05-12T11:57:16.943 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2021-05-12T11:57:16.998 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2021-05-12T11:57:17.003 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2021-05-12T11:57:17.130 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2021-05-12T11:57:17.324 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:57:17.549 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:57:18.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:57:18.020 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2021-05-12T11:57:26.626 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:04 ago on Wed 12 May 2021 11:56:59 AM UTC. 2021-05-12T11:57:28.523 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:57:28.523 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2021-05-12T11:57:28.524 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:28.524 INFO:teuthology.orchestra.run.smithi191.stdout: 2021-05-12T11:57:28.524 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2021-05-12T11:57:28.608 DEBUG:teuthology.parallel:result is None 2021-05-12T11:57:28.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo lsb_release -is 2021-05-12T11:57:28.648 INFO:teuthology.orchestra.run.smithi191.stdout:RedHatEnterprise 2021-05-12T11:57:28.650 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-05-12T11:57:28.650 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=36c2cf2fb22a9790b2c097e158985773b35f1740 2021-05-12T11:57:28.805 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-05-12T11:57:28.824 INFO:teuthology.orchestra.run.smithi191.stdout:17.0.0-4087.g36c2cf2f.el8 2021-05-12T11:57:28.825 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-4087.g36c2cf2f.el8 2021-05-12T11:57:28.825 INFO:teuthology.task.install:The correct ceph version 17.0.0-4087.g36c2cf2f is installed. 2021-05-12T11:57:28.826 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-05-12T11:57:28.827 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2021-05-12T11:57:28.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-05-12T11:57:28.888 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-05-12T11:57:28.889 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2021-05-12T11:57:28.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2021-05-12T11:57:28.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-05-12T11:57:29.016 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-05-12T11:57:29.016 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2021-05-12T11:57:29.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2021-05-12T11:57:29.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-05-12T11:57:29.144 INFO:teuthology.run_tasks:Running task workunit... 2021-05-12T11:57:29.162 INFO:tasks.workunit:Pulling workunits from ref 5cd0ad0001da189274219f5e829752717eb856d2 2021-05-12T11:57:29.163 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-05-12T11:57:29.163 INFO:tasks.workunit:timeout=3h 2021-05-12T11:57:29.164 INFO:tasks.workunit:cleanup=True 2021-05-12T11:57:29.164 DEBUG:teuthology.orchestra.run.smithi191:> stat -- /home/ubuntu/cephtest/mnt.0 2021-05-12T11:57:29.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-05-12T11:57:29.199 INFO:teuthology.orchestra.run.smithi191.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-05-12T11:57:29.200 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-05-12T11:57:29.253 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-05-12T11:57:29.253 DEBUG:teuthology.orchestra.run.smithi191:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-05-12T11:57:29.307 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5cd0ad0001da189274219f5e829752717eb856d2 2021-05-12T11:57:29.361 INFO:tasks.workunit.client.0.smithi191.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-05-12T11:58:05.859 INFO:tasks.workunit.client.0.smithi191.stderr:Note: switching to '5cd0ad0001da189274219f5e829752717eb856d2'. 2021-05-12T11:58:05.859 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T11:58:05.860 INFO:tasks.workunit.client.0.smithi191.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-05-12T11:58:05.860 INFO:tasks.workunit.client.0.smithi191.stderr:changes and commit them, and you can discard any commits you make in this 2021-05-12T11:58:05.860 INFO:tasks.workunit.client.0.smithi191.stderr:state without impacting any branches by switching back to a branch. 2021-05-12T11:58:05.860 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T11:58:05.860 INFO:tasks.workunit.client.0.smithi191.stderr:If you want to create a new branch to retain commits you create, you may 2021-05-12T11:58:05.861 INFO:tasks.workunit.client.0.smithi191.stderr:do so (now or later) by using -c with the switch command. Example: 2021-05-12T11:58:05.861 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T11:58:05.861 INFO:tasks.workunit.client.0.smithi191.stderr: git switch -c 2021-05-12T11:58:05.861 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T11:58:05.862 INFO:tasks.workunit.client.0.smithi191.stderr:Or undo this operation with: 2021-05-12T11:58:05.862 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T11:58:05.862 INFO:tasks.workunit.client.0.smithi191.stderr: git switch - 2021-05-12T11:58:05.862 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T11:58:05.862 INFO:tasks.workunit.client.0.smithi191.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2021-05-12T11:58:05.863 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T11:58:05.863 INFO:tasks.workunit.client.0.smithi191.stderr:HEAD is now at 5cd0ad0001d Merge branch 'negative_progress' of https://github.com/dvanders/ceph into wip-kefu-testing-2021-05-05-1449 2021-05-12T11:58:05.864 DEBUG:teuthology.orchestra.run.smithi191:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-05-12T11:58:05.920 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2021-05-12T11:58:05.920 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-05-12T11:58:05.973 INFO:tasks.workunit:Running workunits matching osd on client.0... 2021-05-12T11:58:05.974 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2021-05-12T11:58:05.974 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2021-05-12T11:58:06.048 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T11:58:06.079 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T11:58:06.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T11:58:06.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T11:58:06.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T11:58:06.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T11:58:06.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T11:58:06.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T11:58:06.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T11:58:06.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T11:58:06.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/bad-inc-map 2021-05-12T11:58:06.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2021-05-12T11:58:06.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2021-05-12T11:58:06.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:56359 2021-05-12T11:58:06.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:56359 2021-05-12T11:58:06.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2021-05-12T11:58:06.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2021-05-12T11:58:06.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none ' 2021-05-12T11:58:06.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:56359 ' 2021-05-12T11:58:06.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2021-05-12T11:58:06.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2021-05-12T11:58:06.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T11:58:06.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2021-05-12T11:58:06.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2021-05-12T11:58:06.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2021-05-12T11:58:06.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2021-05-12T11:58:06.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2021-05-12T11:58:06.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-05-12T11:58:06.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T11:58:06.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-05-12T11:58:06.089 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:58:06.089 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:58:06.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:58:06.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:58:06.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:58:06.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:58:06.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T11:58:06.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T11:58:06.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T11:58:06.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T11:58:06.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T11:58:06.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T11:58:06.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:06.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T11:58:06.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T11:58:06.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:06.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T11:58:06.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T11:58:06.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-05-12T11:58:06.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T11:58:06.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:06.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:06.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29357 2021-05-12T11:58:06.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T11:58:06.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T11:58:06.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2021-05-12T11:58:06.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T11:58:06.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:06.100 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:06.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29357 2021-05-12T11:58:06.100 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T11:58:06.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T11:58:06.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T11:58:06.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T11:58:06.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2021-05-12T11:58:06.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2021-05-12T11:58:06.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2021-05-12T11:58:06.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2021-05-12T11:58:06.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2021-05-12T11:58:06.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T11:58:06.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T11:58:06.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T11:58:06.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2021-05-12T11:58:06.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2021-05-12T11:58:06.561 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T11:58:06.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:06.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:06.562 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:06.562 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:06.563 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:06.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29357/$cluster-$name.asok' 2021-05-12T11:58:06.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T11:58:06.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T11:58:06.841 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T11:58:06.841 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T11:58:06.841 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T11:58:06.841 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T11:58:06.842 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T11:58:06.842 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T11:58:06.842 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T11:58:06.842 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T11:58:06.843 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:58:06.843 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:06.843 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:06.843 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29357/ceph-mon.a.asok 2021-05-12T11:58:06.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T11:58:06.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29357/ceph-mon.a.asok config get fsid 2021-05-12T11:58:06.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T11:58:06.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T11:58:06.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T11:58:06.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T11:58:06.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T11:58:06.938 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T11:58:06.939 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T11:58:06.939 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T11:58:06.939 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:58:06.939 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:06.939 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:06.940 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29357/ceph-mon.a.asok 2021-05-12T11:58:06.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T11:58:06.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29357/ceph-mon.a.asok config get mon_host 2021-05-12T11:58:07.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2021-05-12T11:58:07.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2021-05-12T11:58:07.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T11:58:07.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T11:58:07.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T11:58:07.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2021-05-12T11:58:07.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T11:58:07.901 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T11:58:07.901 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:07.902 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:07.902 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:07.902 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:07.902 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:07.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29357/$cluster-$name.asok' 2021-05-12T11:58:07.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T11:58:07.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T11:58:07.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2021-05-12T11:58:07.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-05-12T11:58:07.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:58:07.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T11:58:07.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:58:07.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2021-05-12T11:58:07.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none --mon-host=127.0.0.1:56359 ' 2021-05-12T11:58:07.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:58:07.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:58:07.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:58:07.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2021-05-12T11:58:07.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2021-05-12T11:58:07.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:58:07.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:58:07.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-05-12T11:58:07.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:58:07.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:07.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:07.930 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:07.930 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:07.930 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:07.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29357/$cluster-$name.asok' 2021-05-12T11:58:07.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' 2021-05-12T11:58:07.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:58:07.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:58:07.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:58:07.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-05-12T11:58:07.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-05-12T11:58:07.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:58:07.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:58:07.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:07.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:58:07.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:58:07.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/bad-inc-map/0 2021-05-12T11:58:07.933 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:58:07.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=481fb43c-dd4a-45af-82a0-12507b7d6128 2021-05-12T11:58:07.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 481fb43c-dd4a-45af-82a0-12507b7d6128' 2021-05-12T11:58:07.934 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 481fb43c-dd4a-45af-82a0-12507b7d6128 2021-05-12T11:58:07.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:58:07.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDPwptgXYvTNxAAr0xe4DsbyHvRPX6JPvdHAA== 2021-05-12T11:58:07.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDPwptgXYvTNxAAr0xe4DsbyHvRPX6JPvdHAA=="}' 2021-05-12T11:58:07.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 481fb43c-dd4a-45af-82a0-12507b7d6128 -i td/bad-inc-map/0/new.json 2021-05-12T11:58:08.256 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:58:08.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/bad-inc-map/0/new.json 2021-05-12T11:58:08.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none --mon-host=127.0.0.1:56359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDPwptgXYvTNxAAr0xe4DsbyHvRPX6JPvdHAA== --osd-uuid 481fb43c-dd4a-45af-82a0-12507b7d6128 2021-05-12T11:58:08.288 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:08.285+0000 7f6b078410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:08.290 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:08.287+0000 7f6b078410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:08.291 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:08.289+0000 7f6b078410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:08.292 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:08.289+0000 7f6b078410c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-05-12T11:58:08.292 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:08.289+0000 7f6b078410c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-05-12T11:58:08.293 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:08.289+0000 7f6b078410c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-05-12T11:58:08.331 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:08.329+0000 7f6b078410c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2021-05-12T11:58:13.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/bad-inc-map/0/keyring 2021-05-12T11:58:13.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:58:13.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T11:58:13.044 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T11:58:13.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:58:13.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T11:58:13.269 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T11:58:13.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none --mon-host=127.0.0.1:56359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:13.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:58:13.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:58:13.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:58:13.289 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:13.286+0000 7f5cf95660c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:13.293 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:13.290+0000 7f5cf95660c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:13.293 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:13.291+0000 7f5cf95660c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:13.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T11:58:13.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:58:13.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T11:58:13.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:58:13.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:58:13.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:13.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:58:13.489 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:58:13.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:13.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:58:13.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:58:13.809 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:13.807+0000 7f5cf95660c0 -1 Falling back to public interface 2021-05-12T11:58:14.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:58:14.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:14.699 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T11:58:14.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:58:14.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:14.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:58:15.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:58:16.015 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:16.012+0000 7f5cf95660c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T11:58:16.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:58:16.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:16.065 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T11:58:16.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:58:16.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:16.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:58:16.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:58:17.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:58:17.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:17.496 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T11:58:17.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T11:58:17.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:17.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:58:17.860 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30418,v1:127.0.0.1:6803/30418] [v2:127.0.0.1:6804/30418,v1:127.0.0.1:6805/30418] exists,up 481fb43c-dd4a-45af-82a0-12507b7d6128 2021-05-12T11:58:17.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:58:17.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:58:17.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:58:17.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2021-05-12T11:58:17.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-05-12T11:58:17.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:58:17.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T11:58:17.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:58:17.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2021-05-12T11:58:17.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none --mon-host=127.0.0.1:56359 ' 2021-05-12T11:58:17.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:58:17.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:58:17.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:58:17.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2021-05-12T11:58:17.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2021-05-12T11:58:17.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:58:17.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:58:17.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-05-12T11:58:17.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:58:17.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:17.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:17.869 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:17.869 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:17.869 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:17.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29357/$cluster-$name.asok' 2021-05-12T11:58:17.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' 2021-05-12T11:58:17.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:58:17.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:58:17.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:58:17.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-05-12T11:58:17.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-05-12T11:58:17.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:58:17.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:58:17.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:17.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:58:17.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:58:17.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/bad-inc-map/1 2021-05-12T11:58:17.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:58:17.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f7214bbb-62c0-4060-84c2-0631feda60df 2021-05-12T11:58:17.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 f7214bbb-62c0-4060-84c2-0631feda60df' 2021-05-12T11:58:17.873 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 f7214bbb-62c0-4060-84c2-0631feda60df 2021-05-12T11:58:17.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:58:17.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDZwptgSvAUNBAAj6S1EvfRxP+jwYUfR2X4Jg== 2021-05-12T11:58:17.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDZwptgSvAUNBAAj6S1EvfRxP+jwYUfR2X4Jg=="}' 2021-05-12T11:58:17.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f7214bbb-62c0-4060-84c2-0631feda60df -i td/bad-inc-map/1/new.json 2021-05-12T11:58:18.374 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T11:58:18.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/bad-inc-map/1/new.json 2021-05-12T11:58:18.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none --mon-host=127.0.0.1:56359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDZwptgSvAUNBAAj6S1EvfRxP+jwYUfR2X4Jg== --osd-uuid f7214bbb-62c0-4060-84c2-0631feda60df 2021-05-12T11:58:18.406 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:18.404+0000 7f7e9530f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:18.408 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:18.406+0000 7f7e9530f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:18.410 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:18.407+0000 7f7e9530f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:18.410 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:18.408+0000 7f7e9530f0c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-05-12T11:58:18.411 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:18.408+0000 7f7e9530f0c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-05-12T11:58:18.411 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:18.408+0000 7f7e9530f0c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-05-12T11:58:18.454 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:18.453+0000 7f7e9530f0c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2021-05-12T11:58:23.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/bad-inc-map/1/keyring 2021-05-12T11:58:23.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:58:23.088 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T11:58:23.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T11:58:23.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:58:23.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T11:58:23.572 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T11:58:23.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none --mon-host=127.0.0.1:56359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:23.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:58:23.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:58:23.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:58:23.591 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:23.588+0000 7f1f743bf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:23.592 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:23.590+0000 7f1f743bf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:23.593 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:23.591+0000 7f1f743bf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:23.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T11:58:23.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:58:23.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T11:58:23.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:58:23.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:58:23.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:23.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:58:23.933 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:58:23.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:23.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:58:24.109 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:24.107+0000 7f1f743bf0c0 -1 Falling back to public interface 2021-05-12T11:58:24.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:58:25.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:58:25.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:25.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:58:25.289 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T11:58:25.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:25.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:58:25.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:58:26.348 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:26.346+0000 7f1f743bf0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T11:58:26.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:58:26.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:26.652 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T11:58:26.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:58:26.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:26.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:58:27.055 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/31891,v1:127.0.0.1:6811/31891] [v2:127.0.0.1:6812/31891,v1:127.0.0.1:6813/31891] exists,up f7214bbb-62c0-4060-84c2-0631feda60df 2021-05-12T11:58:27.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:58:27.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:58:27.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:58:27.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2021-05-12T11:58:27.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-05-12T11:58:27.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:58:27.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T11:58:27.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:58:27.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2021-05-12T11:58:27.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none --mon-host=127.0.0.1:56359 ' 2021-05-12T11:58:27.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:58:27.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:58:27.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:58:27.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2021-05-12T11:58:27.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2021-05-12T11:58:27.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:58:27.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:58:27.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-05-12T11:58:27.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:58:27.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:27.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:27.065 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:27.065 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:27.065 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:27.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29357/$cluster-$name.asok' 2021-05-12T11:58:27.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' 2021-05-12T11:58:27.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:58:27.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:58:27.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:58:27.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-05-12T11:58:27.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-05-12T11:58:27.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:58:27.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:58:27.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:27.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:58:27.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:58:27.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/bad-inc-map/2 2021-05-12T11:58:27.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:58:27.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=36596672-c7da-4e5d-b783-da5b71204474 2021-05-12T11:58:27.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 36596672-c7da-4e5d-b783-da5b71204474' 2021-05-12T11:58:27.069 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 36596672-c7da-4e5d-b783-da5b71204474 2021-05-12T11:58:27.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:58:27.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDjwptgRR05BBAAVVWbyt859ea5EMLrRmydkQ== 2021-05-12T11:58:27.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDjwptgRR05BBAAVVWbyt859ea5EMLrRmydkQ=="}' 2021-05-12T11:58:27.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 36596672-c7da-4e5d-b783-da5b71204474 -i td/bad-inc-map/2/new.json 2021-05-12T11:58:27.639 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T11:58:27.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/bad-inc-map/2/new.json 2021-05-12T11:58:27.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none --mon-host=127.0.0.1:56359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDjwptgRR05BBAAVVWbyt859ea5EMLrRmydkQ== --osd-uuid 36596672-c7da-4e5d-b783-da5b71204474 2021-05-12T11:58:27.671 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:27.668+0000 7ff1352630c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:27.730 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:27.728+0000 7ff1352630c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:27.732 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:27.729+0000 7ff1352630c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:27.732 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:27.730+0000 7ff1352630c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-05-12T11:58:27.733 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:27.730+0000 7ff1352630c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-05-12T11:58:27.733 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:27.730+0000 7ff1352630c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-05-12T11:58:27.788 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:27.787+0000 7ff1352630c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2021-05-12T11:58:32.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/bad-inc-map/2/keyring 2021-05-12T11:58:32.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:58:32.780 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T11:58:32.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T11:58:32.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:58:33.264 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T11:58:33.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T11:58:33.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=e584c941-20dc-457c-8d52-cca4cdded1e3 --auth-supported=none --mon-host=127.0.0.1:56359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29357/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:33.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:58:33.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:58:33.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:58:33.283 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:33.280+0000 7f6f02aea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:33.285 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:33.281+0000 7f6f02aea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:33.286 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:33.283+0000 7f6f02aea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:33.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T11:58:33.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:58:33.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T11:58:33.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:58:33.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:58:33.625 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:58:33.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:33.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:58:33.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:33.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:58:33.802 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:33.800+0000 7f6f02aea0c0 -1 Falling back to public interface 2021-05-12T11:58:33.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:58:34.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:58:34.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:34.982 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T11:58:34.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:58:34.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:34.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:58:35.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:58:36.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:58:36.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:36.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:58:36.747 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T11:58:36.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:36.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:58:37.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:58:38.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:58:38.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:38.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T11:58:38.295 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T11:58:38.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:38.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:58:38.920 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:38.917+0000 7f6f02aea0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T11:58:38.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:58:39.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:58:39.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:39.989 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T11:58:39.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T11:58:39.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:39.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:58:40.353 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/33284,v1:127.0.0.1:6819/33284] [v2:127.0.0.1:6820/33284,v1:127.0.0.1:6821/33284] exists,up 36596672-c7da-4e5d-b783-da5b71204474 2021-05-12T11:58:40.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:58:40.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:58:40.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:58:40.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2021-05-12T11:58:40.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2021-05-12T11:58:40.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2021-05-12T11:58:41.263 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foo' already exists 2021-05-12T11:58:41.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T11:58:42.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2021-05-12T11:58:43.133 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 min_size to 1 2021-05-12T11:58:43.142 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2021-05-12T11:58:44.139 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 min_size to 2 2021-05-12T11:58:44.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2021-05-12T11:58:49.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2021-05-12T11:58:49.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2021-05-12T11:58:49.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:58:49.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T11:58:49.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:58:49.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:58:49.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 10 )) 2021-05-12T11:58:49.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:58:49.153 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:58:49.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:49.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:58:49.523 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30418,v1:127.0.0.1:6803/30418] [v2:127.0.0.1:6804/30418,v1:127.0.0.1:6805/30418] exists,up 481fb43c-dd4a-45af-82a0-12507b7d6128 2021-05-12T11:58:49.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:58:49.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:58:49.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:58:49.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2021-05-12T11:58:49.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2021-05-12T11:58:49.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:58:49.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T11:58:49.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:58:49.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:58:49.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 10 )) 2021-05-12T11:58:49.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:58:49.528 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:58:49.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:49.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:58:49.877 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/31891,v1:127.0.0.1:6811/31891] [v2:127.0.0.1:6812/31891,v1:127.0.0.1:6813/31891] exists,up f7214bbb-62c0-4060-84c2-0631feda60df 2021-05-12T11:58:49.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:58:49.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:58:49.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:58:49.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2021-05-12T11:58:49.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2021-05-12T11:58:49.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:58:49.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T11:58:49.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:58:49.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:58:49.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 10 )) 2021-05-12T11:58:49.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:58:49.881 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:58:49.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:49.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:58:50.238 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/33284,v1:127.0.0.1:6819/33284] [v2:127.0.0.1:6820/33284,v1:127.0.0.1:6821/33284] exists,up 36596672-c7da-4e5d-b783-da5b71204474 2021-05-12T11:58:50.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:58:50.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:58:50.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:58:50.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2021-05-12T11:58:50.241 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:40.848+0000 7f6ee79f2700 -1 osd.2 15 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.241 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:41.156+0000 7f6ee99f6700 -1 osd.2 16 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.241 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:41.635+0000 7f6ee99f6700 -1 osd.2 17 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.241 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:41.894+0000 7f6ee99f6700 -1 osd.2 18 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.242 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:42.833+0000 7f6ee79f2700 -1 osd.2 19 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.242 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:43.126+0000 7f6ee99f6700 -1 osd.2 21 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.242 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:43.387+0000 7f6ee99f6700 -1 osd.2 22 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.243 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:43.931+0000 7f6ee79f2700 -1 osd.2 23 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.243 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:44.133+0000 7f6ee99f6700 -1 osd.2 24 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.243 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:44.484+0000 7f6ee99f6700 -1 osd.2 25 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.243 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:44.778+0000 7f6ee99f6700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-05-12T11:58:50.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2021-05-12T11:58:50.244 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:40.848+0000 7f6ee79f2700 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2021-05-12T11:58:50.244 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:41.157+0000 7f6ee99f6700 10 osd.2 16 handle_osd_map bailing because last < start (16<17) 2021-05-12T11:58:50.245 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:41.635+0000 7f6ee99f6700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2021-05-12T11:58:50.245 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:41.894+0000 7f6ee99f6700 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2021-05-12T11:58:50.245 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:42.833+0000 7f6ee79f2700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2021-05-12T11:58:50.245 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:43.127+0000 7f6ee99f6700 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2021-05-12T11:58:50.246 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:43.387+0000 7f6ee99f6700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2021-05-12T11:58:50.246 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:43.932+0000 7f6ee79f2700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2021-05-12T11:58:50.246 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:44.133+0000 7f6ee99f6700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2021-05-12T11:58:50.246 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:44.484+0000 7f6ee99f6700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2021-05-12T11:58:50.246 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T11:58:44.779+0000 7f6ee99f6700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2021-05-12T11:58:50.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2021-05-12T11:58:50.247 INFO:tasks.workunit.client.0.smithi191.stdout:success 2021-05-12T11:58:50.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2021-05-12T11:58:50.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2021-05-12T11:58:50.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2021-05-12T11:58:50.801 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foo' does not exist 2021-05-12T11:58:50.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2021-05-12T11:58:50.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:58:50.812 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:58:50.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:58:50.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:58:50.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:58:50.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:58:50.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2021-05-12T11:58:50.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-05-12T11:58:50.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T11:58:50.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-05-12T11:58:50.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:58:50.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:58:50.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:58:50.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:58:50.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:58:50.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:58:50.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T11:58:50.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T11:58:50.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T11:58:50.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T11:58:50.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T11:58:50.926 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T11:58:50.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:50.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T11:58:50.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T11:58:50.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:50.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T11:58:50.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T11:58:50.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-05-12T11:58:50.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T11:58:50.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:50.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:50.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29357 2021-05-12T11:58:50.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T11:58:50.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T11:58:50.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T11:58:50.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/bad-inc-map 0 2021-05-12T11:58:50.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-05-12T11:58:50.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T11:58:50.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-05-12T11:58:50.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:58:50.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:58:50.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:58:50.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:58:50.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:58:50.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:58:50.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T11:58:50.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T11:58:50.942 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T11:58:50.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T11:58:50.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T11:58:50.943 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T11:58:50.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:50.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T11:58:50.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T11:58:50.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:50.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T11:58:50.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T11:58:50.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-05-12T11:58:50.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T11:58:50.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:50.948 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29357 2021-05-12T11:58:50.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29357 2021-05-12T11:58:50.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T11:58:50.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T11:58:50.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T11:58:50.949 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T11:58:50.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T11:58:51.018 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2021-05-12T11:58:51.018 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2021-05-12T11:58:51.078 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T11:58:51.088 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T11:58:51.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T11:58:51.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T11:58:51.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T11:58:51.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T11:58:51.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T11:58:51.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T11:58:51.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T11:58:51.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T11:58:51.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/divergent-priors 2021-05-12T11:58:51.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2021-05-12T11:58:51.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2021-05-12T11:58:51.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2021-05-12T11:58:51.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2021-05-12T11:58:51.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2021-05-12T11:58:51.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2021-05-12T11:58:51.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2021-05-12T11:58:51.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2021-05-12T11:58:51.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2021-05-12T11:58:51.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2021-05-12T11:58:51.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2021-05-12T11:58:51.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2021-05-12T11:58:51.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2021-05-12T11:58:51.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2021-05-12T11:58:51.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2021-05-12T11:58:51.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2021-05-12T11:58:51.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2021-05-12T11:58:51.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2021-05-12T11:58:51.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2021-05-12T11:58:51.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2021-05-12T11:58:51.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2021-05-12T11:58:51.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2021-05-12T11:58:51.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2021-05-12T11:58:51.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2021-05-12T11:58:51.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2021-05-12T11:58:51.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2021-05-12T11:58:51.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none ' 2021-05-12T11:58:51.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2021-05-12T11:58:51.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2021-05-12T11:58:51.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2021-05-12T11:58:51.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T11:58:51.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2021-05-12T11:58:51.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T11:58:51.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2021-05-12T11:58:51.104 INFO:tasks.workunit.client.0.smithi191.stderr:TEST_divergent_2 2021-05-12T11:58:51.104 INFO:tasks.workunit.client.0.smithi191.stderr:TEST_divergent_3 2021-05-12T11:58:51.104 INFO:tasks.workunit.client.0.smithi191.stderr:TEST_divergent_ec' 2021-05-12T11:58:51.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T11:58:51.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T11:58:51.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T11:58:51.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T11:58:51.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T11:58:51.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T11:58:51.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T11:58:51.106 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:58:51.106 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:58:51.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:58:51.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:58:51.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:58:51.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:58:51.107 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T11:58:51.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T11:58:51.108 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T11:58:51.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T11:58:51.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T11:58:51.108 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T11:58:51.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:51.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T11:58:51.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T11:58:51.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:51.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T11:58:51.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T11:58:51.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T11:58:51.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T11:58:51.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:51.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:58:51.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34183 2021-05-12T11:58:51.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T11:58:51.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T11:58:51.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T11:58:51.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T11:58:51.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:51.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:58:51.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34183 2021-05-12T11:58:51.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T11:58:51.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T11:58:51.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T11:58:51.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T11:58:51.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T11:58:51.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2021-05-12T11:58:51.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2021-05-12T11:58:51.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2021-05-12T11:58:51.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2021-05-12T11:58:51.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2021-05-12T11:58:51.118 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2021-05-12T11:58:51.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2021-05-12T11:58:51.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2021-05-12T11:58:51.120 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T11:58:51.120 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T11:58:51.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2021-05-12T11:58:51.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T11:58:51.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T11:58:51.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T11:58:51.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T11:58:51.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T11:58:51.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T11:58:51.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T11:58:51.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:51.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:51.481 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:51.481 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:51.482 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:58:51.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T11:58:51.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T11:58:51.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T11:58:51.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T11:58:51.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T11:58:51.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T11:58:51.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T11:58:51.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T11:58:51.760 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T11:58:51.760 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T11:58:51.760 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T11:58:51.761 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:58:51.761 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:51.761 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:58:51.761 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-mon.a.asok 2021-05-12T11:58:51.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T11:58:51.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.34183/ceph-mon.a.asok config get fsid 2021-05-12T11:58:51.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T11:58:51.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T11:58:51.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T11:58:51.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T11:58:51.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T11:58:51.855 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T11:58:51.855 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T11:58:51.855 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T11:58:51.855 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:58:51.856 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:51.856 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:58:51.856 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-mon.a.asok 2021-05-12T11:58:51.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T11:58:51.857 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.34183/ceph-mon.a.asok config get mon_host 2021-05-12T11:58:51.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2021-05-12T11:58:51.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T11:58:51.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T11:58:51.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T11:58:51.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T11:58:51.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T11:58:51.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T11:58:52.835 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T11:58:52.835 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:52.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:52.836 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:52.836 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:52.836 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:58:52.837 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T11:58:52.837 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T11:58:52.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T11:58:52.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-05-12T11:58:52.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2021-05-12T11:58:52.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T11:58:52.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:58:52.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T11:58:52.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:58:52.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T11:58:52.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T11:58:52.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:58:52.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:58:52.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:58:52.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T11:58:52.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T11:58:52.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:58:52.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:58:52.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T11:58:52.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:58:52.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:52.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:52.871 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:52.871 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:52.871 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:58:52.871 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T11:58:52.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T11:58:52.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:58:52.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:58:52.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:58:52.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T11:58:52.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T11:58:52.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:58:52.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:58:52.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:52.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:58:52.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:58:52.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T11:58:52.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:58:52.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=076f4795-1efd-4143-b86e-b70ffb6f20f8 2021-05-12T11:58:52.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 076f4795-1efd-4143-b86e-b70ffb6f20f8' 2021-05-12T11:58:52.875 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 076f4795-1efd-4143-b86e-b70ffb6f20f8 2021-05-12T11:58:52.875 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:58:52.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD8wptgh0OnNBAAMFElCsU5yjWoqnvkG4y4jA== 2021-05-12T11:58:52.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD8wptgh0OnNBAAMFElCsU5yjWoqnvkG4y4jA=="}' 2021-05-12T11:58:52.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 076f4795-1efd-4143-b86e-b70ffb6f20f8 -i td/divergent-priors/0/new.json 2021-05-12T11:58:53.198 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:58:53.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T11:58:53.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD8wptgh0OnNBAAMFElCsU5yjWoqnvkG4y4jA== --osd-uuid 076f4795-1efd-4143-b86e-b70ffb6f20f8 2021-05-12T11:58:53.239 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:53.237+0000 7f53c43100c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:53.241 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:53.239+0000 7f53c43100c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:53.243 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:53.240+0000 7f53c43100c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:53.244 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:53.241+0000 7f53c43100c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T11:58:53.244 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:53.241+0000 7f53c43100c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T11:58:53.244 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:53.241+0000 7f53c43100c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T11:58:53.281 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:53.278+0000 7f53c43100c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T11:58:58.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T11:58:58.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:58:58.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T11:58:58.102 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T11:58:58.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:58:58.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T11:58:58.584 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T11:58:58.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:58.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:58:58.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:58:58.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:58:58.612 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:58.609+0000 7f803acc70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:58.614 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:58.612+0000 7f803acc70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:58.615 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:58.613+0000 7f803acc70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:58.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T11:58:58.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:58:58.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T11:58:58.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:58:58.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:58:58.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:58.957 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:58:58.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:58:58.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:58.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:58:59.131 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:58:59.129+0000 7f803acc70c0 -1 Falling back to public interface 2021-05-12T11:58:59.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:00.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:00.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:00.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:59:00.328 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T11:59:00.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:00.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:59:00.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:01.293 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:01.290+0000 7f803acc70c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T11:59:01.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:01.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:01.703 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T11:59:01.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:59:01.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:01.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:59:02.074 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/35310,v1:127.0.0.1:6803/35310] [v2:127.0.0.1:6804/35310,v1:127.0.0.1:6805/35310] exists,up 076f4795-1efd-4143-b86e-b70ffb6f20f8 2021-05-12T11:59:02.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:02.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:02.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:02.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-05-12T11:59:02.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2021-05-12T11:59:02.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T11:59:02.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:59:02.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T11:59:02.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:59:02.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T11:59:02.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T11:59:02.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:59:02.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:59:02.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:59:02.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T11:59:02.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T11:59:02.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:59:02.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:59:02.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T11:59:02.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:59:02.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:59:02.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:59:02.084 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:59:02.084 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:02.084 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:59:02.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T11:59:02.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T11:59:02.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:59:02.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:59:02.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:59:02.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T11:59:02.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T11:59:02.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:59:02.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:59:02.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:02.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:59:02.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:59:02.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T11:59:02.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:59:02.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=1f571db8-2fee-4731-aa84-e47ff0b8961f 2021-05-12T11:59:02.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 1f571db8-2fee-4731-aa84-e47ff0b8961f' 2021-05-12T11:59:02.088 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 1f571db8-2fee-4731-aa84-e47ff0b8961f 2021-05-12T11:59:02.088 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:59:02.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAGw5tgXeejBRAA5ksrfU5DF9VImGitJGArsw== 2021-05-12T11:59:02.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAGw5tgXeejBRAA5ksrfU5DF9VImGitJGArsw=="}' 2021-05-12T11:59:02.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 1f571db8-2fee-4731-aa84-e47ff0b8961f -i td/divergent-priors/1/new.json 2021-05-12T11:59:02.685 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T11:59:02.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T11:59:02.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAGw5tgXeejBRAA5ksrfU5DF9VImGitJGArsw== --osd-uuid 1f571db8-2fee-4731-aa84-e47ff0b8961f 2021-05-12T11:59:02.727 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:02.725+0000 7f0a5cec50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:02.729 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:02.727+0000 7f0a5cec50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:02.731 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:02.728+0000 7f0a5cec50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:02.732 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:02.729+0000 7f0a5cec50c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T11:59:02.732 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:02.729+0000 7f0a5cec50c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T11:59:02.732 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:02.729+0000 7f0a5cec50c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T11:59:02.772 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:02.770+0000 7f0a5cec50c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T11:59:07.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T11:59:07.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:59:07.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T11:59:07.627 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T11:59:07.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:59:08.124 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T11:59:08.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T11:59:08.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:08.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:59:08.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:59:08.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:59:08.153 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:08.150+0000 7f108788d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:08.155 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:08.152+0000 7f108788d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:08.156 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:08.154+0000 7f108788d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:08.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T11:59:08.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:59:08.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T11:59:08.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:59:08.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:59:08.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:08.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:59:08.493 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:59:08.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:08.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:08.677 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:08.675+0000 7f108788d0c0 -1 Falling back to public interface 2021-05-12T11:59:08.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:09.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:09.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:09.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:59:09.908 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T11:59:09.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:09.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:10.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:10.861 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:10.858+0000 7f108788d0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T11:59:11.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:11.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:11.283 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T11:59:11.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:59:11.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:11.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:11.660 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/36695,v1:127.0.0.1:6811/36695] [v2:127.0.0.1:6812/36695,v1:127.0.0.1:6813/36695] exists,up 1f571db8-2fee-4731-aa84-e47ff0b8961f 2021-05-12T11:59:11.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:11.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:11.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:11.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-05-12T11:59:11.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2021-05-12T11:59:11.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T11:59:11.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:59:11.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T11:59:11.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:59:11.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T11:59:11.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T11:59:11.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:59:11.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:59:11.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:59:11.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T11:59:11.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T11:59:11.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:59:11.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:59:11.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T11:59:11.668 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:59:11.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:59:11.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:59:11.669 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:59:11.669 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:11.669 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:59:11.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T11:59:11.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T11:59:11.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:59:11.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:59:11.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:59:11.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T11:59:11.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T11:59:11.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:59:11.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:59:11.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:11.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:59:11.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:59:11.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T11:59:11.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:59:11.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e783e361-c2f1-4c7c-946d-95632561588b 2021-05-12T11:59:11.673 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 e783e361-c2f1-4c7c-946d-95632561588b 2021-05-12T11:59:11.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 e783e361-c2f1-4c7c-946d-95632561588b' 2021-05-12T11:59:11.674 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:59:11.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAPw5tgU3GqKBAA++dzl7cJFC6xjN07hX3LGg== 2021-05-12T11:59:11.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAPw5tgU3GqKBAA++dzl7cJFC6xjN07hX3LGg=="}' 2021-05-12T11:59:11.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e783e361-c2f1-4c7c-946d-95632561588b -i td/divergent-priors/2/new.json 2021-05-12T11:59:12.287 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T11:59:12.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T11:59:12.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAPw5tgU3GqKBAA++dzl7cJFC6xjN07hX3LGg== --osd-uuid e783e361-c2f1-4c7c-946d-95632561588b 2021-05-12T11:59:12.329 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:12.326+0000 7fd3cac770c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:12.330 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:12.328+0000 7fd3cac770c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:12.332 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:12.330+0000 7fd3cac770c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:12.333 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:12.331+0000 7fd3cac770c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T11:59:12.333 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:12.331+0000 7fd3cac770c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T11:59:12.334 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:12.331+0000 7fd3cac770c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T11:59:12.446 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:12.444+0000 7fd3cac770c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T11:59:17.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T11:59:17.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:59:17.260 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T11:59:17.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T11:59:17.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:59:17.751 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T11:59:17.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T11:59:17.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:17.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:59:17.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:59:17.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:59:17.779 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:17.776+0000 7f14792ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:17.800 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:17.797+0000 7f14792ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:17.801 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:17.799+0000 7f14792ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:18.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T11:59:18.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:59:18.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T11:59:18.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:59:18.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:59:18.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:18.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:59:18.127 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T11:59:18.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:18.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:59:18.321 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:18.319+0000 7f14792ff0c0 -1 Falling back to public interface 2021-05-12T11:59:18.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:19.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:19.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:19.488 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T11:59:19.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:59:19.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:19.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:59:19.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:20.552 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T11:59:20.550+0000 7f14792ff0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T11:59:20.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:20.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:20.867 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T11:59:20.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:59:20.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:20.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:59:21.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:22.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:22.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:22.277 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T11:59:22.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T11:59:22.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:22.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:59:22.651 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/38080,v1:127.0.0.1:6819/38080] [v2:127.0.0.1:6820/38080,v1:127.0.0.1:6821/38080] exists,up e783e361-c2f1-4c7c-946d-95632561588b 2021-05-12T11:59:22.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:22.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:22.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:22.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2021-05-12T11:59:23.426 INFO:tasks.workunit.client.0.smithi191.stderr:noout is set 2021-05-12T11:59:23.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2021-05-12T11:59:24.240 INFO:tasks.workunit.client.0.smithi191.stderr:noin is set 2021-05-12T11:59:24.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2021-05-12T11:59:25.059 INFO:tasks.workunit.client.0.smithi191.stderr:nodown is set 2021-05-12T11:59:25.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2021-05-12T11:59:25.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-05-12T11:59:25.779 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test' already exists 2021-05-12T11:59:25.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T11:59:26.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2021-05-12T11:59:27.507 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 size to 3 2021-05-12T11:59:27.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2021-05-12T11:59:28.270 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 min_size to 2 2021-05-12T11:59:28.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2021-05-12T11:59:28.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T11:59:28.281 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T11:59:28.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T11:59:28.642 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T11:59:28.643 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T11:59:28.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T11:59:28.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T11:59:28.644 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T11:59:28.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836488 2021-05-12T11:59:28.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836488 2021-05-12T11:59:28.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488' 2021-05-12T11:59:28.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T11:59:28.777 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T11:59:28.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672966 2021-05-12T11:59:28.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672966 2021-05-12T11:59:28.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2021-05-12T11:59:28.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T11:59:28.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T11:59:29.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509444 2021-05-12T11:59:29.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509444 2021-05-12T11:59:29.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509444' 2021-05-12T11:59:29.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T11:59:29.045 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2021-05-12T11:59:29.045 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T11:59:29.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T11:59:29.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836488 2021-05-12T11:59:29.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T11:59:29.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836488 2021-05-12T11:59:29.047 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836488 2021-05-12T11:59:29.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-05-12T11:59:29.048 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T11:59:29.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836485 -lt 21474836488 2021-05-12T11:59:29.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T11:59:30.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T11:59:30.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T11:59:30.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836488 2021-05-12T11:59:30.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T11:59:30.795 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-05-12T11:59:30.795 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T11:59:30.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T11:59:30.797 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672966 2021-05-12T11:59:30.797 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T11:59:30.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672966 2021-05-12T11:59:30.798 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672966 2021-05-12T11:59:30.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-05-12T11:59:30.799 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T11:59:31.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672966 2021-05-12T11:59:31.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T11:59:31.215 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-05-12T11:59:31.216 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T11:59:31.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T11:59:31.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509444 2021-05-12T11:59:31.218 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T11:59:31.218 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 64424509444 2021-05-12T11:59:31.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509444 2021-05-12T11:59:31.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-05-12T11:59:31.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T11:59:31.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509444 -lt 64424509444 2021-05-12T11:59:31.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2021-05-12T11:59:31.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T11:59:31.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T11:59:31.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T11:59:31.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T11:59:31.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T11:59:31.625 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T11:59:31.625 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T11:59:31.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T11:59:31.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T11:59:31.626 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T11:59:31.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T11:59:31.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T11:59:31.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T11:59:31.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T11:59:31.711 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T11:59:32.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T11:59:32.083 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T11:59:32.083 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T11:59:32.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T11:59:32.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T11:59:32.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T11:59:32.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836490 2021-05-12T11:59:32.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836490 2021-05-12T11:59:32.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490' 2021-05-12T11:59:32.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T11:59:32.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T11:59:32.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672969 2021-05-12T11:59:32.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672969 2021-05-12T11:59:32.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2021-05-12T11:59:32.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T11:59:32.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T11:59:32.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509447 2021-05-12T11:59:32.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509447 2021-05-12T11:59:32.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672969 2-64424509447' 2021-05-12T11:59:32.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T11:59:32.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-05-12T11:59:32.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T11:59:32.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T11:59:32.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836490 2021-05-12T11:59:32.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T11:59:32.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836490 2021-05-12T11:59:32.489 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836490 2021-05-12T11:59:32.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-05-12T11:59:32.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T11:59:32.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836490 2021-05-12T11:59:32.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T11:59:33.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T11:59:33.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T11:59:34.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836490 2021-05-12T11:59:34.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T11:59:34.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-05-12T11:59:34.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T11:59:34.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T11:59:34.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672969 2021-05-12T11:59:34.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T11:59:34.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672969 2021-05-12T11:59:34.226 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672969 2021-05-12T11:59:34.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-05-12T11:59:34.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T11:59:34.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672969 2021-05-12T11:59:34.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T11:59:34.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2021-05-12T11:59:34.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T11:59:34.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T11:59:34.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509447 2021-05-12T11:59:34.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T11:59:34.596 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 64424509447 2021-05-12T11:59:34.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509447 2021-05-12T11:59:34.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-05-12T11:59:34.597 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T11:59:34.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509447 -lt 64424509447 2021-05-12T11:59:34.984 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T11:59:34.984 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T11:59:34.984 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T11:59:35.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T11:59:35.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T11:59:35.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T11:59:35.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T11:59:35.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T11:59:35.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T11:59:35.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T11:59:35.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T11:59:35.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T11:59:35.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T11:59:35.857 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T11:59:35.857 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T11:59:36.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T11:59:36.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T11:59:36.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T11:59:36.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2021-05-12T11:59:36.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2021-05-12T11:59:36.706 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T11:59:36.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2021-05-12T11:59:36.716 INFO:tasks.workunit.client.0.smithi191.stdout:primary and soon to be divergent is 1 2021-05-12T11:59:36.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2021-05-12T11:59:36.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2021-05-12T11:59:37.061 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T11:59:37.062 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T11:59:26.768041+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T11:59:37.062 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T11:59:37.062 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T11:59:37.063 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T11:59:37.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2021-05-12T11:59:37.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-05-12T11:59:37.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2021-05-12T11:59:37.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2021-05-12T11:59:37.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-05-12T11:59:37.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2021-05-12T11:59:37.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2021-05-12T11:59:37.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-05-12T11:59:37.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2021-05-12T11:59:37.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2021-05-12T11:59:37.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2021-05-12T11:59:37.076 INFO:tasks.workunit.client.0.smithi191.stdout:writing initial objects 2021-05-12T11:59:37.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2021-05-12T11:59:37.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:37.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2021-05-12T11:59:37.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:37.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2021-05-12T11:59:37.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:37.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2021-05-12T11:59:37.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:37.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2021-05-12T11:59:37.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:37.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2021-05-12T11:59:37.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:37.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2021-05-12T11:59:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2021-05-12T11:59:37.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:37.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2021-05-12T11:59:38.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:38.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2021-05-12T11:59:38.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:38.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2021-05-12T11:59:38.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:38.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2021-05-12T11:59:38.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:38.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2021-05-12T11:59:38.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:38.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2021-05-12T11:59:38.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:38.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2021-05-12T11:59:38.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:38.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2021-05-12T11:59:39.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:39.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2021-05-12T11:59:39.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:39.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2021-05-12T11:59:39.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:39.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2021-05-12T11:59:39.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:39.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2021-05-12T11:59:39.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:39.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2021-05-12T11:59:39.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:39.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2021-05-12T11:59:39.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:39.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2021-05-12T11:59:39.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:39.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2021-05-12T11:59:40.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:40.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2021-05-12T11:59:40.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:40.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2021-05-12T11:59:40.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:40.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2021-05-12T11:59:40.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:40.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2021-05-12T11:59:40.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:40.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2021-05-12T11:59:40.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:40.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2021-05-12T11:59:40.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:40.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2021-05-12T11:59:41.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:41.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2021-05-12T11:59:41.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:41.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2021-05-12T11:59:41.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:41.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2021-05-12T11:59:41.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:41.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2021-05-12T11:59:41.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:41.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2021-05-12T11:59:41.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:41.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2021-05-12T11:59:41.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:41.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2021-05-12T11:59:42.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:42.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2021-05-12T11:59:42.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:42.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2021-05-12T11:59:42.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:42.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2021-05-12T11:59:42.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:42.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2021-05-12T11:59:42.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:42.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2021-05-12T11:59:42.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:42.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2021-05-12T11:59:42.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:42.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2021-05-12T11:59:42.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:42.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2021-05-12T11:59:43.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:43.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2021-05-12T11:59:43.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:43.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2021-05-12T11:59:43.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:43.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2021-05-12T11:59:43.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:43.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2021-05-12T11:59:43.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:43.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2021-05-12T11:59:43.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:43.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2021-05-12T11:59:43.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:43.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2021-05-12T11:59:44.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:44.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2021-05-12T11:59:44.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:44.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2021-05-12T11:59:44.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:44.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2021-05-12T11:59:44.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:44.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2021-05-12T11:59:44.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:44.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2021-05-12T11:59:44.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:44.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2021-05-12T11:59:44.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:44.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2021-05-12T11:59:44.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:44.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2021-05-12T11:59:45.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:45.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2021-05-12T11:59:45.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:45.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2021-05-12T11:59:45.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:45.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2021-05-12T11:59:45.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:45.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2021-05-12T11:59:45.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:45.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2021-05-12T11:59:45.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:45.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2021-05-12T11:59:45.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:45.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2021-05-12T11:59:45.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:45.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2021-05-12T11:59:46.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:46.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2021-05-12T11:59:46.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:46.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2021-05-12T11:59:46.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:46.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2021-05-12T11:59:46.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:46.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2021-05-12T11:59:46.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:46.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2021-05-12T11:59:46.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:46.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2021-05-12T11:59:46.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:46.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2021-05-12T11:59:47.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:47.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2021-05-12T11:59:47.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:47.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2021-05-12T11:59:47.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:47.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2021-05-12T11:59:47.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:47.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2021-05-12T11:59:47.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:47.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2021-05-12T11:59:47.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:47.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2021-05-12T11:59:47.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:47.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2021-05-12T11:59:47.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:47.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2021-05-12T11:59:48.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:48.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2021-05-12T11:59:48.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:48.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2021-05-12T11:59:48.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:48.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2021-05-12T11:59:48.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:48.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2021-05-12T11:59:48.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:48.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2021-05-12T11:59:48.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:48.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2021-05-12T11:59:48.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:48.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2021-05-12T11:59:49.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:49.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2021-05-12T11:59:49.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:49.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2021-05-12T11:59:49.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:49.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2021-05-12T11:59:49.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:49.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2021-05-12T11:59:49.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:49.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2021-05-12T11:59:49.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:49.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2021-05-12T11:59:49.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:49.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2021-05-12T11:59:49.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:49.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2021-05-12T11:59:50.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:50.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2021-05-12T11:59:50.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T11:59:50.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-05-12T11:59:50.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T11:59:50.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2021-05-12T11:59:50.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T11:59:50.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T11:59:50.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T11:59:50.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T11:59:50.348 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T11:59:50.349 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T11:59:50.349 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T11:59:50.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T11:59:50.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T11:59:50.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T11:59:50.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T11:59:50.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T11:59:50.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T11:59:50.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T11:59:50.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T11:59:50.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T11:59:50.764 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T11:59:50.764 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T11:59:50.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T11:59:50.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T11:59:50.765 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T11:59:50.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836496 2021-05-12T11:59:50.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836496 2021-05-12T11:59:50.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496' 2021-05-12T11:59:50.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T11:59:50.926 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T11:59:51.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672974 2021-05-12T11:59:51.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672974 2021-05-12T11:59:51.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-05-12T11:59:51.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T11:59:51.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T11:59:51.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509452 2021-05-12T11:59:51.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509452 2021-05-12T11:59:51.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509452' 2021-05-12T11:59:51.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T11:59:51.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-05-12T11:59:51.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T11:59:51.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T11:59:51.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836496 2021-05-12T11:59:51.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T11:59:51.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836496 2021-05-12T11:59:51.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-05-12T11:59:51.195 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836496 2021-05-12T11:59:51.196 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T11:59:51.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836494 -lt 21474836496 2021-05-12T11:59:51.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T11:59:52.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T11:59:52.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T11:59:52.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836496 -lt 21474836496 2021-05-12T11:59:52.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T11:59:52.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-05-12T11:59:52.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T11:59:52.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T11:59:52.933 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672974 2021-05-12T11:59:52.933 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T11:59:52.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672974 2021-05-12T11:59:52.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-05-12T11:59:52.934 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672974 2021-05-12T11:59:52.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T11:59:53.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672974 -lt 42949672974 2021-05-12T11:59:53.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T11:59:53.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-05-12T11:59:53.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T11:59:53.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T11:59:53.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509452 2021-05-12T11:59:53.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T11:59:53.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509452 2021-05-12T11:59:53.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-05-12T11:59:53.335 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 64424509452 2021-05-12T11:59:53.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T11:59:53.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509453 -lt 64424509452 2021-05-12T11:59:53.712 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T11:59:53.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T11:59:53.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T11:59:54.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T11:59:54.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T11:59:54.207 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T11:59:54.208 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T11:59:54.208 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T11:59:54.208 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T11:59:54.208 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T11:59:54.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T11:59:54.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T11:59:54.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T11:59:54.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T11:59:54.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T11:59:55.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T11:59:55.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T11:59:55.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T11:59:55.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2021-05-12T11:59:55.048 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T11:59:55.048 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T11:59:55.048 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T11:59:55.049 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T11:59:55.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2021-05-12T11:59:55.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2021-05-12T11:59:55.415 INFO:tasks.workunit.client.0.smithi191.stdout:blackholing osds 0 2 2021-05-12T11:59:55.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2021-05-12T11:59:55.779 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T11:59:55.780 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T11:59:26.768041+0000 29'100 29:113 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T11:59:55.780 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T11:59:55.780 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T11:59:55.781 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T11:59:55.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-05-12T11:59:55.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2021-05-12T11:59:55.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T11:59:55.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T11:59:55.793 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:59:55.793 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:55.793 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:59:55.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.0.asok 2021-05-12T11:59:55.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-05-12T11:59:55.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T11:59:55.883 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T11:59:55.884 INFO:tasks.workunit.client.0.smithi191.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T11:59:55.884 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T11:59:55.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-05-12T11:59:55.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2021-05-12T11:59:55.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T11:59:55.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T11:59:55.895 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:59:55.895 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:55.895 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:59:55.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.2.asok 2021-05-12T11:59:55.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-05-12T11:59:55.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.2.asok config set objectstore_blackhole 1 2021-05-12T11:59:55.984 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T11:59:55.985 INFO:tasks.workunit.client.0.smithi191.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T11:59:55.985 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T11:59:55.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2021-05-12T11:59:55.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2021-05-12T11:59:55.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2021-05-12T11:59:55.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2021-05-12T11:59:55.995 INFO:tasks.workunit.client.0.smithi191.stdout:writing divergent object 2021-05-12T11:59:55.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2021-05-12T11:59:55.996 INFO:tasks.workunit.client.0.smithi191.stdout:create missing divergent object 2021-05-12T11:59:55.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-05-12T11:59:55.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2021-05-12T11:59:55.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local pooltype=rep 2021-05-12T11:59:55.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2021-05-12T11:59:55.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local which=data 2021-05-12T11:59:55.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2021-05-12T11:59:55.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local poolname=test 2021-05-12T11:59:56.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2021-05-12T11:59:56.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: local objname=existing_99 2021-05-12T11:59:56.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: shift 2021-05-12T11:59:56.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local dir=td/divergent-priors 2021-05-12T11:59:56.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: shift 2021-05-12T11:59:56.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: local shard_id=0 2021-05-12T11:59:56.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: shift 2021-05-12T11:59:56.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-05-12T11:59:56.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: get_osds test existing_99 2021-05-12T11:59:56.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local poolname=test 2021-05-12T11:59:56.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: get_osds: local objectname=existing_99 2021-05-12T11:59:56.002 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: ceph --format json osd map test existing_99 2021-05-12T11:59:56.002 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: jq '.acting | .[]' 2021-05-12T11:59:56.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local 'osds=1 2021-05-12T11:59:56.365 INFO:tasks.workunit.client.0.smithi191.stderr:0 2021-05-12T11:59:56.365 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T11:59:56.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: get_osds: echo 1 0 2 2021-05-12T11:59:56.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local -a initial_osds 2021-05-12T11:59:56.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local osd_id=1 2021-05-12T11:59:56.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: '[' rep '!=' ec ']' 2021-05-12T11:59:56.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: shard_id= 2021-05-12T11:59:56.368 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: cat td/divergent-priors/1/type 2021-05-12T11:59:56.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: type=bluestore 2021-05-12T11:59:56.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-05-12T11:59:56.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local daemon=osd 2021-05-12T11:59:56.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local id=1 2021-05-12T11:59:56.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local config=bluestore_debug_inject_read_err 2021-05-12T11:59:56.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local value=true 2021-05-12T11:59:56.370 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: jq 'has("success")' 2021-05-12T11:59:56.370 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: get_asok_path osd.1 2021-05-12T11:59:56.370 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T11:59:56.371 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T11:59:56.371 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:59:56.371 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:56.371 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:59:56.371 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.1.asok 2021-05-12T11:59:56.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.34183/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-05-12T11:59:56.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: test true == true 2021-05-12T11:59:56.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: local loop=0 2021-05-12T11:59:56.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: grep -q Invalid 2021-05-12T11:59:56.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: get_asok_path osd.1 2021-05-12T11:59:56.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T11:59:56.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T11:59:56.471 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:59:56.471 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:56.471 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T11:59:56.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.1.asok 2021-05-12T11:59:56.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: CEPH_ARGS= 2021-05-12T11:59:56.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.1.asok injectdataerr test existing_99 2021-05-12T11:59:56.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2021-05-12T11:59:56.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2021-05-12T12:00:06.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2021-05-12T12:00:06.597 INFO:tasks.workunit.client.0.smithi191.stderr:rados: no process found 2021-05-12T12:00:06.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2021-05-12T12:00:06.598 INFO:tasks.workunit.client.0.smithi191.stdout:killing all the osds 2021-05-12T12:00:06.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2021-05-12T12:00:06.950 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:00:06.951 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2021-05-12T12:00:00.100242+0000 29'101 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:00:06.951 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:00:06.951 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:00:06.952 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:00:06.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2021-05-12T12:00:06.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:00:06.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:00:06.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:00:06.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:00:06.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:00:07.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:00:07.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-05-12T12:00:07.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2021-05-12T12:00:07.656 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already down. 2021-05-12T12:00:07.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-05-12T12:00:07.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2021-05-12T12:00:08.204 INFO:tasks.workunit.client.0.smithi191.stderr:osd.1 is already down. 2021-05-12T12:00:08.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-05-12T12:00:08.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2021-05-12T12:00:08.718 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already down. 2021-05-12T12:00:08.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:08.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2021-05-12T12:00:09.272 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already out. 2021-05-12T12:00:09.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:09.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2021-05-12T12:00:09.896 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already out. 2021-05-12T12:00:09.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2021-05-12T12:00:09.905 INFO:tasks.workunit.client.0.smithi191.stdout:bringing up non_divergent 0 2 2021-05-12T12:00:09.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2021-05-12T12:00:10.253 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:00:10.254 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2021-05-12T12:00:00.100242+0000 29'101 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:00:10.254 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:00:10.254 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:00:10.255 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:00:10.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:10.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2021-05-12T12:00:10.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:00:10.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:00:10.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:00:10.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:00:10.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:00:10.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:00:10.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:00:10.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:00:10.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:00:10.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:00:10.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:00:10.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:00:10.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:00:10.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:00:10.272 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:00:10.273 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:00:10.273 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:00:10.273 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:00:10.273 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:10.274 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:00:10.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:00:10.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:00:10.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:00:10.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:00:10.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:00:10.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:00:10.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:00:10.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:10.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:00:10.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:00:10.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:00:10.277 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:00:10.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:00:10.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:10.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:00:10.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:00:10.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:00:10.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:00:10.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:00:10.294 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:10.292+0000 7f933055e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:10.296 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:10.294+0000 7f933055e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:10.298 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:10.295+0000 7f933055e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:10.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:00:10.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:00:10.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:00:10.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:00:10.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:00:10.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:10.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:00:10.639 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:00:10.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:10.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:00:10.820 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:10.817+0000 7f933055e0c0 -1 Falling back to public interface 2021-05-12T12:00:11.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:12.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:12.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:12.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:00:12.037 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:00:12.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:12.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:00:12.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:13.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:13.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:13.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:00:13.418 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:00:13.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:13.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:00:13.613 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:13.610+0000 7f933055e0c0 -1 osd.0 29 log_to_monitors {default=true} 2021-05-12T12:00:13.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:14.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:14.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:14.793 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:00:14.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:00:14.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:14.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:00:15.165 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/42157,v1:127.0.0.1:6803/42157] [v2:127.0.0.1:6804/42157,v1:127.0.0.1:6805/42157] exists,up 076f4795-1efd-4143-b86e-b70ffb6f20f8 2021-05-12T12:00:15.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:00:15.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:00:15.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:00:15.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:15.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2021-05-12T12:00:15.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:00:15.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:00:15.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:00:15.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:00:15.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:00:15.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:00:15.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:00:15.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:00:15.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:00:15.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:00:15.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:00:15.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:00:15.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:00:15.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:00:15.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:00:15.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:00:15.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:00:15.174 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:00:15.175 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:15.175 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:00:15.175 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:00:15.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:00:15.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:00:15.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:00:15.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:00:15.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:00:15.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:00:15.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:15.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:00:15.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:00:15.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:00:15.178 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:00:15.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:00:15.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:15.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:00:15.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:00:15.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:00:15.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:00:15.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:00:15.195 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:15.193+0000 7f23c03b00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:15.197 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:15.195+0000 7f23c03b00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:15.199 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:15.196+0000 7f23c03b00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:15.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:00:15.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:00:15.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:00:15.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:00:15.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:00:15.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:15.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:00:15.573 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:00:15.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:15.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:15.721 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:15.718+0000 7f23c03b00c0 -1 Falling back to public interface 2021-05-12T12:00:15.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:16.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:16.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:16.957 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:00:16.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:00:16.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:16.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:17.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:18.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:18.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:18.338 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:00:18.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:00:18.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:18.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:18.495 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:18.492+0000 7f23c03b00c0 -1 osd.2 29 log_to_monitors {default=true} 2021-05-12T12:00:18.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:19.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:19.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:19.716 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:00:19.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:00:19.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:19.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:20.091 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/42740,v1:127.0.0.1:6811/42740] [v2:127.0.0.1:6812/42740,v1:127.0.0.1:6813/42740] exists,up e783e361-c2f1-4c7c-946d-95632561588b 2021-05-12T12:00:20.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:00:20.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:00:20.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:00:20.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:20.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2021-05-12T12:00:20.608 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already in. 2021-05-12T12:00:20.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:20.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2021-05-12T12:00:21.516 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already in. 2021-05-12T12:00:21.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:00:21.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2021-05-12T12:00:21.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:00:21.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:00:21.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:00:21.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:00:21.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:00:21.528 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:00:21.528 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:00:21.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:00:21.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:00:21.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:00:21.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:00:21.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:00:21.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:00:21.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:00:21.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:00:21.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:00:21.968 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:00:21.968 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:00:21.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:00:21.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:21.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:00:22.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=154618822660 2021-05-12T12:00:22.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 154618822660 2021-05-12T12:00:22.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-154618822660' 2021-05-12T12:00:22.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:22.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:00:22.200 INFO:tasks.workunit.client.0.smithi191.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:00:22.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:00:22.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:00:22.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:00:22.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:22.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:00:22.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=163208757251 2021-05-12T12:00:22.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 163208757251 2021-05-12T12:00:22.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2021-05-12T12:00:22.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:22.339 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-154618822660 2021-05-12T12:00:22.339 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:22.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:00:22.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-154618822660 2021-05-12T12:00:22.342 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:22.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=154618822660 2021-05-12T12:00:22.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2021-05-12T12:00:22.343 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 154618822660 2021-05-12T12:00:22.344 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:00:22.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 154618822658 -lt 154618822660 2021-05-12T12:00:22.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:00:23.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:00:23.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:00:24.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 154618822660 -lt 154618822660 2021-05-12T12:00:24.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:24.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-163208757251 2021-05-12T12:00:24.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:24.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:00:24.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-163208757251 2021-05-12T12:00:24.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:24.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=163208757251 2021-05-12T12:00:24.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2021-05-12T12:00:24.080 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 163208757251 2021-05-12T12:00:24.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:00:24.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 163208757251 -lt 163208757251 2021-05-12T12:00:24.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:00:24.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:24.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:24.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:00:24.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:24.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:24.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:24.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:24.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:24.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:24.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:25.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:00:25.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:25.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:25.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:25.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:00:25.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:00:25.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:00:25.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:00:25.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:00:25.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:00:25.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:00:25.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:25.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:25.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:25.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:25.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:25.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:25.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:26.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:00:26.226 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:26.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:26.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:26.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:00:26.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:00:26.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:00:26.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:00:26.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:00:26.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:00:26.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:00:26.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:00:26.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:00:27.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:00:27.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:00:27.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:00:27.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:00:27.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:00:27.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:00:27.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:27.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:27.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:27.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:27.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:27.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:27.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:27.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:00:27.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:27.795 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:27.795 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:28.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:00:28.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:00:28.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:00:28.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:00:28.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:00:28.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:00:28.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:00:28.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:00:28.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:00:28.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:00:28.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:00:28.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:00:28.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:00:28.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:00:29.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:00:29.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:29.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:29.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:29.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:29.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:29.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:29.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:29.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:00:29.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:29.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:29.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:29.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:00:29.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:00:29.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:00:29.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:00:29.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:00:30.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:00:30.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:00:30.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:00:30.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:00:30.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:00:30.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:00:30.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:00:30.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:00:30.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:00:31.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:00:31.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:31.281 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:31.282 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:31.282 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:31.282 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:31.282 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:31.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:31.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:00:31.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:31.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:31.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:32.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:00:32.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:00:32.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:00:32.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:00:32.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:00:32.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:00:32.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:00:32.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:00:32.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:00:32.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:00:32.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:00:32.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:00:32.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:00:32.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:00:34.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:00:34.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:34.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:34.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:34.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:34.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:34.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:34.211 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:34.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:00:34.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:34.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:34.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:35.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:00:35.056 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:00:35.056 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:00:35.056 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:00:35.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:00:35.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:00:35.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:00:35.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:00:35.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:00:35.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:00:35.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:00:35.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:00:35.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:00:35.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:00:38.739 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:00:38.739 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:38.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:38.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:38.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:38.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:38.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:38.742 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:39.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:00:39.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:39.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:39.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:39.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:00:39.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:00:39.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:00:39.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:00:39.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:00:39.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:00:39.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:00:39.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:00:39.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:00:40.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:00:40.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:00:40.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:00:40.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:00:40.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:00:46.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:00:46.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:46.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:46.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:46.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:46.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:46.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:46.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:46.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:00:46.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:46.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:46.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:47.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:00:47.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:00:47.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:00:47.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:00:47.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:00:47.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:00:47.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:00:47.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:00:47.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:00:47.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:00:47.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:00:47.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:00:47.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:00:47.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:00:55.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:00:55.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:55.133 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:55.133 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:55.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:55.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:55.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:55.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:55.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:00:55.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:55.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:55.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:55.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:00:55.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:00:55.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:00:55.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:00:55.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:00:55.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:00:55.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:00:55.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:00:55.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:00:56.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:00:56.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:00:56.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:00:56.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:00:56.925 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:00:56.926 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_fingerprint": "0f86e764-37b6-4a91-a2cd-9a2448f2d9b5", 2021-05-12T12:00:56.926 INFO:tasks.workunit.client.0.smithi191.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:00:56.926 INFO:tasks.workunit.client.0.smithi191.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:00:56.926 INFO:tasks.workunit.client.0.smithi191.stdout: "timestamp": "2021-05-12T12:00:56.922595+0000", 2021-05-12T12:00:56.927 INFO:tasks.workunit.client.0.smithi191.stdout: "tag": "", 2021-05-12T12:00:56.927 INFO:tasks.workunit.client.0.smithi191.stdout: "health": { 2021-05-12T12:00:56.927 INFO:tasks.workunit.client.0.smithi191.stdout: "status": "HEALTH_ERR", 2021-05-12T12:00:56.927 INFO:tasks.workunit.client.0.smithi191.stdout: "checks": { 2021-05-12T12:00:56.927 INFO:tasks.workunit.client.0.smithi191.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:00:56.928 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:00:56.928 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:00:56.928 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:00:56.928 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:00:56.928 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.929 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:00:56.929 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.929 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:00:56.929 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.930 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:56.930 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:00:56.930 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.930 INFO:tasks.workunit.client.0.smithi191.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:00:56.930 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:00:56.931 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:00:56.931 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:00:56.931 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 17 2021-05-12T12:00:56.931 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.932 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [], 2021-05-12T12:00:56.932 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:00:56.932 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.932 INFO:tasks.workunit.client.0.smithi191.stdout: "OSD_DOWN": { 2021-05-12T12:00:56.932 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:00:56.933 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:00:56.933 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 osds down", 2021-05-12T12:00:56.933 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:00:56.933 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.934 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:00:56.934 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.934 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "osd.1 (root=default,host=smithi191) is down" 2021-05-12T12:00:56.934 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.934 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:56.935 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:00:56.935 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.935 INFO:tasks.workunit.client.0.smithi191.stdout: "PG_DEGRADED": { 2021-05-12T12:00:56.935 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:00:56.936 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:00:56.936 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2021-05-12T12:00:56.936 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:00:56.936 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.936 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:00:56.937 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.937 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:00:56.937 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.937 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:56.938 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:00:56.938 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.938 INFO:tasks.workunit.client.0.smithi191.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:00:56.938 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:00:56.938 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:00:56.939 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:00:56.939 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:00:56.939 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.939 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:00:56.939 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.940 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:00:56.940 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.940 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.940 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:00:56.940 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.941 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:56.941 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:00:56.941 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.941 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.941 INFO:tasks.workunit.client.0.smithi191.stdout: "mutes": [] 2021-05-12T12:00:56.942 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.942 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_first_committed": 1, 2021-05-12T12:00:56.942 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_last_committed": 1, 2021-05-12T12:00:56.942 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap": { 2021-05-12T12:00:56.942 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:00:56.943 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:00:56.943 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T11:58:51.137539Z", 2021-05-12T12:00:56.943 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T11:58:51.137539Z", 2021-05-12T12:00:56.943 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release": 17, 2021-05-12T12:00:56.943 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:00:56.944 INFO:tasks.workunit.client.0.smithi191.stdout: "election_strategy": 1, 2021-05-12T12:00:56.944 INFO:tasks.workunit.client.0.smithi191.stdout: "disallowed_leaders: ": "", 2021-05-12T12:00:56.944 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": false, 2021-05-12T12:00:56.944 INFO:tasks.workunit.client.0.smithi191.stdout: "features": { 2021-05-12T12:00:56.945 INFO:tasks.workunit.client.0.smithi191.stdout: "persistent": [ 2021-05-12T12:00:56.945 INFO:tasks.workunit.client.0.smithi191.stdout: "kraken", 2021-05-12T12:00:56.945 INFO:tasks.workunit.client.0.smithi191.stdout: "luminous", 2021-05-12T12:00:56.945 INFO:tasks.workunit.client.0.smithi191.stdout: "mimic", 2021-05-12T12:00:56.945 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap-prune", 2021-05-12T12:00:56.946 INFO:tasks.workunit.client.0.smithi191.stdout: "nautilus", 2021-05-12T12:00:56.946 INFO:tasks.workunit.client.0.smithi191.stdout: "octopus", 2021-05-12T12:00:56.946 INFO:tasks.workunit.client.0.smithi191.stdout: "pacific", 2021-05-12T12:00:56.946 INFO:tasks.workunit.client.0.smithi191.stdout: "elector-pinging", 2021-05-12T12:00:56.946 INFO:tasks.workunit.client.0.smithi191.stdout: "quincy" 2021-05-12T12:00:56.947 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:56.947 INFO:tasks.workunit.client.0.smithi191.stdout: "optional": [] 2021-05-12T12:00:56.947 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.947 INFO:tasks.workunit.client.0.smithi191.stdout: "mons": [ 2021-05-12T12:00:56.947 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.948 INFO:tasks.workunit.client.0.smithi191.stdout: "rank": 0, 2021-05-12T12:00:56.948 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "a", 2021-05-12T12:00:56.948 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:00:56.948 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:56.948 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.948 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:56.949 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:00:56.949 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 0 2021-05-12T12:00:56.949 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.949 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.949 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.950 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:00:56.950 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:00:56.950 INFO:tasks.workunit.client.0.smithi191.stdout: "priority": 0, 2021-05-12T12:00:56.950 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 0, 2021-05-12T12:00:56.950 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_location": "{}" 2021-05-12T12:00:56.951 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.951 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.951 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.951 INFO:tasks.workunit.client.0.smithi191.stdout: "quorum": [ 2021-05-12T12:00:56.952 INFO:tasks.workunit.client.0.smithi191.stdout: 0 2021-05-12T12:00:56.952 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:56.952 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap": { 2021-05-12T12:00:56.952 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 43, 2021-05-12T12:00:56.952 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:00:56.953 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T11:58:52.052167+0000", 2021-05-12T12:00:56.953 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:00:21.587090+0000", 2021-05-12T12:00:56.953 INFO:tasks.workunit.client.0.smithi191.stdout: "last_up_change": "2021-05-12T12:00:18.704172+0000", 2021-05-12T12:00:56.953 INFO:tasks.workunit.client.0.smithi191.stdout: "last_in_change": "2021-05-12T12:00:21.041539+0000", 2021-05-12T12:00:56.953 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:00:56.954 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_num": 5800384, 2021-05-12T12:00:56.954 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_set": [ 2021-05-12T12:00:56.954 INFO:tasks.workunit.client.0.smithi191.stdout: "nodown", 2021-05-12T12:00:56.954 INFO:tasks.workunit.client.0.smithi191.stdout: "noin", 2021-05-12T12:00:56.955 INFO:tasks.workunit.client.0.smithi191.stdout: "noout", 2021-05-12T12:00:56.955 INFO:tasks.workunit.client.0.smithi191.stdout: "pglog_hardlimit", 2021-05-12T12:00:56.955 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snapdirs", 2021-05-12T12:00:56.955 INFO:tasks.workunit.client.0.smithi191.stdout: "recovery_deletes", 2021-05-12T12:00:56.955 INFO:tasks.workunit.client.0.smithi191.stdout: "sortbitwise" 2021-05-12T12:00:56.956 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:56.956 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_version": 9, 2021-05-12T12:00:56.956 INFO:tasks.workunit.client.0.smithi191.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:00:56.956 INFO:tasks.workunit.client.0.smithi191.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:00:56.956 INFO:tasks.workunit.client.0.smithi191.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:00:56.957 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_snapshot": "", 2021-05-12T12:00:56.957 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_max": 1, 2021-05-12T12:00:56.957 INFO:tasks.workunit.client.0.smithi191.stdout: "max_osd": 3, 2021-05-12T12:00:56.957 INFO:tasks.workunit.client.0.smithi191.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:00:56.957 INFO:tasks.workunit.client.0.smithi191.stdout: "min_compat_client": "jewel", 2021-05-12T12:00:56.958 INFO:tasks.workunit.client.0.smithi191.stdout: "require_osd_release": "quincy", 2021-05-12T12:00:56.958 INFO:tasks.workunit.client.0.smithi191.stdout: "pools": [ 2021-05-12T12:00:56.958 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.958 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:00:56.958 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_name": "test", 2021-05-12T12:00:56.959 INFO:tasks.workunit.client.0.smithi191.stdout: "create_time": "2021-05-12T11:59:25.347528+0000", 2021-05-12T12:00:56.959 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": 1, 2021-05-12T12:00:56.959 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_names": "hashpspool", 2021-05-12T12:00:56.959 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:00:56.959 INFO:tasks.workunit.client.0.smithi191.stdout: "size": 3, 2021-05-12T12:00:56.960 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 2, 2021-05-12T12:00:56.960 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_rule": 0, 2021-05-12T12:00:56.960 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:00:56.960 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:00:56.960 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:00:56.961 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:00:56.961 INFO:tasks.workunit.client.0.smithi191.stdout: "object_hash": 2, 2021-05-12T12:00:56.961 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:00:56.961 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num": 1, 2021-05-12T12:00:56.961 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num": 1, 2021-05-12T12:00:56.962 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num_target": 1, 2021-05-12T12:00:56.962 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_target": 1, 2021-05-12T12:00:56.962 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_pending": 1, 2021-05-12T12:00:56.962 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pg_merge_meta": { 2021-05-12T12:00:56.962 INFO:tasks.workunit.client.0.smithi191.stdout: "source_pgid": "0.0", 2021-05-12T12:00:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: "ready_epoch": 0, 2021-05-12T12:00:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_started": 0, 2021-05-12T12:00:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": 0, 2021-05-12T12:00:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: "source_version": "0'0", 2021-05-12T12:00:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: "target_version": "0'0" 2021-05-12T12:00:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: "last_change": "29", 2021-05-12T12:00:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend": "0", 2021-05-12T12:00:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:00:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:00:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: "auid": 0, 2021-05-12T12:00:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:00:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_seq": 0, 2021-05-12T12:00:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_epoch": 0, 2021-05-12T12:00:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_snaps": [], 2021-05-12T12:00:56.966 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps": "[]", 2021-05-12T12:00:56.966 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_bytes": 0, 2021-05-12T12:00:56.966 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_objects": 0, 2021-05-12T12:00:56.966 INFO:tasks.workunit.client.0.smithi191.stdout: "tiers": [], 2021-05-12T12:00:56.966 INFO:tasks.workunit.client.0.smithi191.stdout: "tier_of": -1, 2021-05-12T12:00:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "read_tier": -1, 2021-05-12T12:00:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "write_tier": -1, 2021-05-12T12:00:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_mode": "none", 2021-05-12T12:00:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_bytes": 0, 2021-05-12T12:00:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_objects": 0, 2021-05-12T12:00:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:00:56.968 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:00:56.968 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:00:56.968 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_flush_age": 0, 2021-05-12T12:00:56.968 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_evict_age": 0, 2021-05-12T12:00:56.968 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profile": "", 2021-05-12T12:00:56.969 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_params": { 2021-05-12T12:00:56.969 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "none" 2021-05-12T12:00:56.969 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.969 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_period": 0, 2021-05-12T12:00:56.969 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_count": 0, 2021-05-12T12:00:56.970 INFO:tasks.workunit.client.0.smithi191.stdout: "use_gmt_hitset": true, 2021-05-12T12:00:56.970 INFO:tasks.workunit.client.0.smithi191.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:00:56.970 INFO:tasks.workunit.client.0.smithi191.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:00:56.970 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:00:56.970 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:00:56.971 INFO:tasks.workunit.client.0.smithi191.stdout: "grade_table": [], 2021-05-12T12:00:56.971 INFO:tasks.workunit.client.0.smithi191.stdout: "stripe_width": 0, 2021-05-12T12:00:56.971 INFO:tasks.workunit.client.0.smithi191.stdout: "expected_num_objects": 0, 2021-05-12T12:00:56.971 INFO:tasks.workunit.client.0.smithi191.stdout: "fast_read": false, 2021-05-12T12:00:56.971 INFO:tasks.workunit.client.0.smithi191.stdout: "options": {}, 2021-05-12T12:00:56.972 INFO:tasks.workunit.client.0.smithi191.stdout: "application_metadata": {} 2021-05-12T12:00:56.972 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.972 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:56.972 INFO:tasks.workunit.client.0.smithi191.stdout: "osds": [ 2021-05-12T12:00:56.972 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.973 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:00:56.973 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "076f4795-1efd-4143-b86e-b70ffb6f20f8", 2021-05-12T12:00:56.973 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:00:56.973 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:00:56.973 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:00:56.974 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:00:56.974 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 5, 2021-05-12T12:00:56.974 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 29, 2021-05-12T12:00:56.974 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 36, 2021-05-12T12:00:56.974 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 41, 2021-05-12T12:00:56.975 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 30, 2021-05-12T12:00:56.975 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:00:56.975 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:00:56.975 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:56.975 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.976 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:56.976 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:00:56.976 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42157 2021-05-12T12:00:56.976 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.976 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.977 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:56.977 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:00:56.977 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42157 2021-05-12T12:00:56.977 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.977 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.978 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.978 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:00:56.978 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:56.978 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.978 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:56.979 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:00:56.979 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42157 2021-05-12T12:00:56.979 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.979 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.979 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:56.980 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:00:56.980 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42157 2021-05-12T12:00:56.980 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.980 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.980 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.980 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:00:56.981 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:56.981 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.981 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:56.981 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:00:56.981 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42157 2021-05-12T12:00:56.982 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.982 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.982 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:56.982 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:00:56.982 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42157 2021-05-12T12:00:56.983 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.983 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.983 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.983 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:00:56.983 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:56.984 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.984 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:56.984 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:00:56.984 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42157 2021-05-12T12:00:56.984 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.985 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.985 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:56.985 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:00:56.985 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42157 2021-05-12T12:00:56.985 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.986 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.986 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.986 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6803/42157", 2021-05-12T12:00:56.986 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6805/42157", 2021-05-12T12:00:56.986 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6809/42157", 2021-05-12T12:00:56.987 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6807/42157", 2021-05-12T12:00:56.987 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:00:56.987 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:00:56.987 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:00:56.987 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.988 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.988 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.988 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:00:56.988 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "1f571db8-2fee-4731-aa84-e47ff0b8961f", 2021-05-12T12:00:56.988 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 0, 2021-05-12T12:00:56.989 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:00:56.989 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:00:56.989 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:00:56.989 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 0, 2021-05-12T12:00:56.989 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 0, 2021-05-12T12:00:56.990 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 10, 2021-05-12T12:00:56.990 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 23, 2021-05-12T12:00:56.990 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 31, 2021-05-12T12:00:56.990 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:00:56.990 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:00:56.991 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:56.991 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.991 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:56.991 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:00:56.991 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 36695 2021-05-12T12:00:56.992 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.992 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.992 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:56.992 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:00:56.992 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 36695 2021-05-12T12:00:56.993 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.993 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.993 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.993 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:00:56.993 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:56.994 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.994 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:56.994 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:00:56.994 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 36695 2021-05-12T12:00:56.994 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.995 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.995 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:56.995 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:00:56.995 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 36695 2021-05-12T12:00:56.995 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.996 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.996 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.996 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:00:56.996 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:56.996 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.997 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:56.997 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:00:56.997 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 36695 2021-05-12T12:00:56.997 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.997 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:56.998 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:56.998 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:00:56.998 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 36695 2021-05-12T12:00:56.998 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:56.999 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:56.999 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:56.999 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:00:56.999 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:56.999 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.000 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:57.000 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:00:57.000 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 36695 2021-05-12T12:00:57.000 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.000 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.001 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:57.001 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:00:57.001 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 36695 2021-05-12T12:00:57.001 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.001 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.002 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.002 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/36695", 2021-05-12T12:00:57.002 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/36695", 2021-05-12T12:00:57.002 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/36695", 2021-05-12T12:00:57.002 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/36695", 2021-05-12T12:00:57.003 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:00:57.003 INFO:tasks.workunit.client.0.smithi191.stdout: "exists" 2021-05-12T12:00:57.003 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.003 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.004 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.004 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:00:57.004 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "e783e361-c2f1-4c7c-946d-95632561588b", 2021-05-12T12:00:57.004 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:00:57.004 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:00:57.005 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:00:57.005 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:00:57.005 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 15, 2021-05-12T12:00:57.005 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 29, 2021-05-12T12:00:57.005 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 38, 2021-05-12T12:00:57.006 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 0, 2021-05-12T12:00:57.006 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 32, 2021-05-12T12:00:57.006 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:00:57.006 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:00:57.006 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:57.007 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.007 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:57.007 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:00:57.007 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42740 2021-05-12T12:00:57.007 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.008 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.008 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:57.008 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:00:57.008 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42740 2021-05-12T12:00:57.008 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.009 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.009 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.009 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:00:57.009 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:57.009 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.010 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:57.010 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:00:57.010 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42740 2021-05-12T12:00:57.010 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.010 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.011 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:57.011 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:00:57.011 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42740 2021-05-12T12:00:57.011 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.011 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.012 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.012 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:00:57.012 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:57.012 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.012 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:57.013 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:00:57.013 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42740 2021-05-12T12:00:57.013 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.013 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.013 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:57.014 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:00:57.014 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42740 2021-05-12T12:00:57.014 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.014 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.014 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.015 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:00:57.015 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:00:57.015 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.015 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:00:57.015 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:00:57.015 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42740 2021-05-12T12:00:57.016 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.016 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.016 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:00:57.016 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:00:57.017 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 42740 2021-05-12T12:00:57.017 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.017 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.017 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.017 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/42740", 2021-05-12T12:00:57.018 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/42740", 2021-05-12T12:00:57.018 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/42740", 2021-05-12T12:00:57.018 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/42740", 2021-05-12T12:00:57.018 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:00:57.018 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:00:57.019 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:00:57.019 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.019 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.019 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.019 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_xinfo": [ 2021-05-12T12:00:57.020 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.020 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:00:57.020 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:00:07.426579+0000", 2021-05-12T12:00:57.020 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:00:57.021 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:00:57.021 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:00:57.021 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:00:57.021 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T11:59:02.328516+0000", 2021-05-12T12:00:57.021 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:00:57.022 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.022 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.022 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:00:57.022 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:00:08.010937+0000", 2021-05-12T12:00:57.022 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:00:57.023 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:00:57.023 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:00:57.023 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:00:57.023 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T11:59:11.854014+0000", 2021-05-12T12:00:57.023 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:00:57.024 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.024 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.024 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:00:57.024 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:00:08.523615+0000", 2021-05-12T12:00:57.024 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:00:57.025 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:00:57.025 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:00:57.025 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:00:57.025 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T11:59:21.600002+0000", 2021-05-12T12:00:57.025 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:00:57.026 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.026 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.026 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap": [], 2021-05-12T12:00:57.026 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap_items": [], 2021-05-12T12:00:57.026 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_temp": [], 2021-05-12T12:00:57.027 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_temp": [], 2021-05-12T12:00:57.027 INFO:tasks.workunit.client.0.smithi191.stdout: "blocklist": {}, 2021-05-12T12:00:57.027 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profiles": { 2021-05-12T12:00:57.027 INFO:tasks.workunit.client.0.smithi191.stdout: "default": { 2021-05-12T12:00:57.027 INFO:tasks.workunit.client.0.smithi191.stdout: "k": "2", 2021-05-12T12:00:57.027 INFO:tasks.workunit.client.0.smithi191.stdout: "m": "2", 2021-05-12T12:00:57.028 INFO:tasks.workunit.client.0.smithi191.stdout: "plugin": "jerasure", 2021-05-12T12:00:57.028 INFO:tasks.workunit.client.0.smithi191.stdout: "technique": "reed_sol_van" 2021-05-12T12:00:57.028 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.028 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.029 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps_queue": [], 2021-05-12T12:00:57.029 INFO:tasks.workunit.client.0.smithi191.stdout: "new_removed_snaps": [], 2021-05-12T12:00:57.029 INFO:tasks.workunit.client.0.smithi191.stdout: "new_purged_snaps": [], 2021-05-12T12:00:57.029 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_node_flags": {}, 2021-05-12T12:00:57.029 INFO:tasks.workunit.client.0.smithi191.stdout: "device_class_flags": {}, 2021-05-12T12:00:57.030 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": { 2021-05-12T12:00:57.030 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_enabled": false, 2021-05-12T12:00:57.030 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_bucket_count": 0, 2021-05-12T12:00:57.030 INFO:tasks.workunit.client.0.smithi191.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:00:57.030 INFO:tasks.workunit.client.0.smithi191.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:00:57.031 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_bucket": 0 2021-05-12T12:00:57.031 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.031 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.031 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_metadata": [ 2021-05-12T12:00:57.031 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.032 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:00:57.032 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:00:57.032 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6804/42157,v1:127.0.0.1:6805/42157]", 2021-05-12T12:00:57.032 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:00:57.032 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:00:57.033 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:00:57.033 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:00:57.033 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:00:57.033 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:00:57.033 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:00:57.034 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:00:57.034 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:00:57.034 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:00:57.034 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:00:57.034 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:00:57.035 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:00:57.035 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:00:57.035 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:00:57.035 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:00:57.035 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:00:57.036 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:00:57.036 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:00:57.036 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:00:57.036 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:00:57.036 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:00:57.037 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:00:57.037 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:00:57.037 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:00:57.037 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6802/42157,v1:127.0.0.1:6803/42157]", 2021-05-12T12:00:57.038 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:00:57.038 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/42157,v1:127.0.0.1:6809/42157]", 2021-05-12T12:00:57.038 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/42157,v1:127.0.0.1:6807/42157]", 2021-05-12T12:00:57.038 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:00:57.039 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:00:57.039 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:00:57.039 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:00:57.039 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:00:57.039 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:00:57.040 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:00:57.040 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:00:57.040 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:00:57.040 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:00:57.041 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:00:57.041 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:00:57.041 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:00:57.041 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.041 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.042 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:00:57.042 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:00:57.042 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/36695,v1:127.0.0.1:6813/36695]", 2021-05-12T12:00:57.042 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:00:57.042 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:00:57.043 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:00:57.043 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:00:57.043 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:00:57.043 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:00:57.044 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:00:57.044 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:00:57.044 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:00:57.044 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:00:57.044 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:00:57.045 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:00:57.045 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:00:57.045 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:00:57.045 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:00:57.045 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:00:57.046 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:00:57.046 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:00:57.046 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:00:57.046 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:00:57.047 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:00:57.047 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:00:57.047 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:00:57.047 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:00:57.047 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:00:57.048 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/36695,v1:127.0.0.1:6811/36695]", 2021-05-12T12:00:57.048 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:00:57.048 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/36695,v1:127.0.0.1:6817/36695]", 2021-05-12T12:00:57.048 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/36695,v1:127.0.0.1:6815/36695]", 2021-05-12T12:00:57.049 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:00:57.049 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:00:57.049 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:00:57.049 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:00:57.050 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:00:57.050 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:00:57.050 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:00:57.050 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:00:57.050 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:00:57.051 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:00:57.051 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:00:57.051 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:00:57.051 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:00:57.052 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.052 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.052 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:00:57.052 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:00:57.053 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/42740,v1:127.0.0.1:6813/42740]", 2021-05-12T12:00:57.053 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:00:57.053 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:00:57.053 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:00:57.053 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:00:57.054 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:00:57.054 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:00:57.054 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:00:57.054 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:00:57.054 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:00:57.055 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:00:57.055 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:00:57.055 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:00:57.055 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:00:57.056 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:00:57.056 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:00:57.056 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:00:57.056 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:00:57.056 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:00:57.057 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:00:57.057 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:00:57.057 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:00:57.057 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:00:57.058 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:00:57.058 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:00:57.058 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:00:57.058 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/42740,v1:127.0.0.1:6811/42740]", 2021-05-12T12:00:57.059 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:00:57.059 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/42740,v1:127.0.0.1:6817/42740]", 2021-05-12T12:00:57.059 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/42740,v1:127.0.0.1:6815/42740]", 2021-05-12T12:00:57.059 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:00:57.060 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:00:57.060 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:00:57.060 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:00:57.060 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:00:57.060 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:00:57.061 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:00:57.061 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:00:57.061 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:00:57.061 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:00:57.062 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:00:57.062 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:00:57.062 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:00:57.062 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.062 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.063 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_clean_epochs": { 2021-05-12T12:00:57.063 INFO:tasks.workunit.client.0.smithi191.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:00:57.063 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": { 2021-05-12T12:00:57.063 INFO:tasks.workunit.client.0.smithi191.stdout: "per_pool": [] 2021-05-12T12:00:57.063 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.064 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_epochs": [ 2021-05-12T12:00:57.064 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.064 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:00:57.064 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 36 2021-05-12T12:00:57.064 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.064 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.065 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:00:57.065 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 10 2021-05-12T12:00:57.065 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.065 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.065 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:00:57.066 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 38 2021-05-12T12:00:57.066 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.066 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.066 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.066 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_first_committed": 1, 2021-05-12T12:00:57.067 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_last_committed": 43, 2021-05-12T12:00:57.067 INFO:tasks.workunit.client.0.smithi191.stdout: "crushmap": { 2021-05-12T12:00:57.067 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": [ 2021-05-12T12:00:57.067 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.067 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:00:57.067 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.0", 2021-05-12T12:00:57.068 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:00:57.068 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.068 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.068 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:00:57.068 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.1", 2021-05-12T12:00:57.069 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:00:57.069 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.069 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.069 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:00:57.069 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.2", 2021-05-12T12:00:57.070 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:00:57.070 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.070 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.070 INFO:tasks.workunit.client.0.smithi191.stdout: "types": [ 2021-05-12T12:00:57.070 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.070 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 0, 2021-05-12T12:00:57.071 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd" 2021-05-12T12:00:57.071 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.071 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.071 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:00:57.071 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "host" 2021-05-12T12:00:57.072 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.072 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.072 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 2, 2021-05-12T12:00:57.072 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "chassis" 2021-05-12T12:00:57.072 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.073 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.073 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 3, 2021-05-12T12:00:57.073 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "rack" 2021-05-12T12:00:57.073 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.073 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.074 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 4, 2021-05-12T12:00:57.074 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "row" 2021-05-12T12:00:57.074 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.074 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.074 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 5, 2021-05-12T12:00:57.075 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pdu" 2021-05-12T12:00:57.075 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.075 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.075 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 6, 2021-05-12T12:00:57.076 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pod" 2021-05-12T12:00:57.076 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.076 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.076 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 7, 2021-05-12T12:00:57.076 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "room" 2021-05-12T12:00:57.077 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.077 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.077 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 8, 2021-05-12T12:00:57.077 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "datacenter" 2021-05-12T12:00:57.077 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.078 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.078 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 9, 2021-05-12T12:00:57.078 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "zone" 2021-05-12T12:00:57.078 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.078 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.079 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 10, 2021-05-12T12:00:57.079 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "region" 2021-05-12T12:00:57.079 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.079 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.079 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:00:57.079 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "root" 2021-05-12T12:00:57.080 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.080 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.080 INFO:tasks.workunit.client.0.smithi191.stdout: "buckets": [ 2021-05-12T12:00:57.080 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.081 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -1, 2021-05-12T12:00:57.081 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default", 2021-05-12T12:00:57.081 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:00:57.081 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:00:57.081 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:00:57.082 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:00:57.082 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:00:57.082 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:00:57.082 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.082 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:00:57.083 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:00:57.083 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:00:57.083 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.083 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.083 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.084 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.084 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -2, 2021-05-12T12:00:57.084 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default~hdd", 2021-05-12T12:00:57.084 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:00:57.084 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:00:57.085 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:00:57.085 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:00:57.085 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:00:57.085 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:00:57.085 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.086 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:00:57.086 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:00:57.086 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:00:57.086 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.086 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.087 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.087 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.087 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:00:57.087 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191", 2021-05-12T12:00:57.087 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:00:57.087 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:00:57.088 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:00:57.088 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:00:57.088 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:00:57.088 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:00:57.088 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.089 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:00:57.089 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:00:57.089 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:00:57.089 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.089 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.090 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:00:57.090 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:00:57.090 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:00:57.090 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.090 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.091 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:00:57.091 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:00:57.091 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:00:57.091 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.091 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.092 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.092 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.092 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:00:57.092 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191~hdd", 2021-05-12T12:00:57.092 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:00:57.093 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:00:57.093 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:00:57.093 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:00:57.093 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:00:57.093 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:00:57.094 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.094 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:00:57.094 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:00:57.094 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:00:57.094 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.095 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.095 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:00:57.095 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:00:57.095 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:00:57.095 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.096 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.096 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:00:57.096 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:00:57.096 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:00:57.096 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.097 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.097 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.097 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.097 INFO:tasks.workunit.client.0.smithi191.stdout: "rules": [ 2021-05-12T12:00:57.097 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.098 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 0, 2021-05-12T12:00:57.098 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "replicated_rule", 2021-05-12T12:00:57.098 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 0, 2021-05-12T12:00:57.098 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:00:57.098 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 1, 2021-05-12T12:00:57.098 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 10, 2021-05-12T12:00:57.099 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2021-05-12T12:00:57.099 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.099 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2021-05-12T12:00:57.099 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2021-05-12T12:00:57.099 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2021-05-12T12:00:57.100 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.100 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.100 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_firstn", 2021-05-12T12:00:57.100 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2021-05-12T12:00:57.100 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2021-05-12T12:00:57.101 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.101 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.101 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2021-05-12T12:00:57.101 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.101 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:00:57.102 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.102 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.102 INFO:tasks.workunit.client.0.smithi191.stdout: "tunables": { 2021-05-12T12:00:57.102 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_tries": 0, 2021-05-12T12:00:57.102 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:00:57.103 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_total_tries": 50, 2021-05-12T12:00:57.103 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:00:57.103 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:00:57.103 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_stable": 1, 2021-05-12T12:00:57.103 INFO:tasks.workunit.client.0.smithi191.stdout: "straw_calc_version": 1, 2021-05-12T12:00:57.103 INFO:tasks.workunit.client.0.smithi191.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:00:57.104 INFO:tasks.workunit.client.0.smithi191.stdout: "profile": "jewel", 2021-05-12T12:00:57.104 INFO:tasks.workunit.client.0.smithi191.stdout: "optimal_tunables": 1, 2021-05-12T12:00:57.104 INFO:tasks.workunit.client.0.smithi191.stdout: "legacy_tunables": 0, 2021-05-12T12:00:57.104 INFO:tasks.workunit.client.0.smithi191.stdout: "minimum_required_version": "jewel", 2021-05-12T12:00:57.104 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables": 1, 2021-05-12T12:00:57.105 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables2": 1, 2021-05-12T12:00:57.105 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v2_rules": 0, 2021-05-12T12:00:57.105 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables3": 1, 2021-05-12T12:00:57.105 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v3_rules": 0, 2021-05-12T12:00:57.105 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v4_buckets": 1, 2021-05-12T12:00:57.106 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables5": 1, 2021-05-12T12:00:57.106 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v5_rules": 0 2021-05-12T12:00:57.106 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.106 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_args": {} 2021-05-12T12:00:57.107 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.107 INFO:tasks.workunit.client.0.smithi191.stdout: "fsmap": { 2021-05-12T12:00:57.107 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:00:57.107 INFO:tasks.workunit.client.0.smithi191.stdout: "default_fscid": -1, 2021-05-12T12:00:57.107 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": { 2021-05-12T12:00:57.108 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": {}, 2021-05-12T12:00:57.108 INFO:tasks.workunit.client.0.smithi191.stdout: "ro_compat": {}, 2021-05-12T12:00:57.108 INFO:tasks.workunit.client.0.smithi191.stdout: "incompat": { 2021-05-12T12:00:57.108 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_1": "base v0.20", 2021-05-12T12:00:57.109 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:00:57.109 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:00:57.109 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:00:57.109 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:00:57.109 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:00:57.110 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_8": "no anchor table", 2021-05-12T12:00:57.110 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_9": "file layout v2", 2021-05-12T12:00:57.110 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:00:57.110 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.110 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.111 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_flags": { 2021-05-12T12:00:57.111 INFO:tasks.workunit.client.0.smithi191.stdout: "enable_multiple": true, 2021-05-12T12:00:57.111 INFO:tasks.workunit.client.0.smithi191.stdout: "ever_enabled_multiple": true 2021-05-12T12:00:57.111 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.111 INFO:tasks.workunit.client.0.smithi191.stdout: "standbys": [], 2021-05-12T12:00:57.112 INFO:tasks.workunit.client.0.smithi191.stdout: "filesystems": [] 2021-05-12T12:00:57.112 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.112 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:00:57.112 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:00:57.112 INFO:tasks.workunit.client.0.smithi191.stdout: "auth": { 2021-05-12T12:00:57.113 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:00:57.113 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 5, 2021-05-12T12:00:57.113 INFO:tasks.workunit.client.0.smithi191.stdout: "num_secrets": 3 2021-05-12T12:00:57.113 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.113 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:00:57.113 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_active": 1, 2021-05-12T12:00:57.114 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_unknown": 0, 2021-05-12T12:00:57.114 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osd": 3, 2021-05-12T12:00:57.114 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_sum": { 2021-05-12T12:00:57.114 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:00:57.114 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48300, 2021-05-12T12:00:57.115 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:00:57.115 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:00:57.115 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:00:57.115 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:00:57.115 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:00:57.116 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:00:57.116 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:00:57.116 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:00:57.116 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:00:57.116 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:00:57.117 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:00:57.117 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:00:57.117 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 100, 2021-05-12T12:00:57.117 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 100, 2021-05-12T12:00:57.117 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:00:57.117 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:00:57.118 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:00:57.118 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:00:57.118 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:00:57.118 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:00:57.119 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:00:57.119 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:00:57.119 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:00:57.119 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:00:57.119 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:00:57.120 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:00:57.120 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:00:57.120 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:00:57.120 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:00:57.120 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:00:57.120 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:00:57.121 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:00:57.121 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:00:57.121 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:00:57.121 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:00:57.121 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:00:57.122 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:00:57.122 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:00:57.122 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:00:57.122 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.122 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:00:57.123 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:00:57.123 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:00:57.123 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:00:57.123 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 0, 2021-05-12T12:00:57.123 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 0, 2021-05-12T12:00:57.123 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:00:57.124 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:00:57.124 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:00:57.124 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:00:57.124 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:00:57.124 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.125 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 16, 2021-05-12T12:00:57.125 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 16, 2021-05-12T12:00:57.125 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 2, 2021-05-12T12:00:57.125 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 2, 2021-05-12T12:00:57.125 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 0 2021-05-12T12:00:57.126 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.126 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum": { 2021-05-12T12:00:57.126 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:00:57.126 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:00:57.126 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:00:57.127 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:00:57.127 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:00:57.127 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:00:57.127 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:00:57.127 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 18492, 2021-05-12T12:00:57.128 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2144, 2021-05-12T12:00:57.128 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:00:57.128 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16317, 2021-05-12T12:00:57.128 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314554308, 2021-05-12T12:00:57.128 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:00:57.129 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:00:57.129 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322103611392, 2021-05-12T12:00:57.129 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:00:57.129 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2195456, 2021-05-12T12:00:57.129 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 429342, 2021-05-12T12:00:57.129 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:00:57.130 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:00:57.130 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:00:57.130 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:00:57.130 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 16709520 2021-05-12T12:00:57.130 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.131 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:00:57.131 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:00:57.131 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:00:57.131 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 1, 2021-05-12T12:00:57.131 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:00:57.132 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:00:57.132 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:00:57.132 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.132 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:00:57.132 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 39, 2021-05-12T12:00:57.132 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 39, 2021-05-12T12:00:57.133 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 39000000, 2021-05-12T12:00:57.133 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 39000000 2021-05-12T12:00:57.133 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.133 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:00:57.133 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:00:57.134 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.134 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum_by_class": { 2021-05-12T12:00:57.134 INFO:tasks.workunit.client.0.smithi191.stdout: "hdd": { 2021-05-12T12:00:57.134 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:00:57.134 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:00:57.135 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:00:57.135 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:00:57.135 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:00:57.135 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:00:57.135 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:00:57.136 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 18492, 2021-05-12T12:00:57.136 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2144, 2021-05-12T12:00:57.136 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:00:57.136 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16317, 2021-05-12T12:00:57.136 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314554308, 2021-05-12T12:00:57.137 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:00:57.137 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:00:57.137 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322103611392, 2021-05-12T12:00:57.137 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:00:57.137 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2195456, 2021-05-12T12:00:57.138 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 429342, 2021-05-12T12:00:57.138 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:00:57.138 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:00:57.138 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:00:57.138 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:00:57.138 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 16709520 2021-05-12T12:00:57.139 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.139 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:00:57.139 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:00:57.139 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:00:57.139 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 1, 2021-05-12T12:00:57.140 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:00:57.140 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:00:57.140 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:00:57.140 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.140 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:00:57.141 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 39, 2021-05-12T12:00:57.141 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 39, 2021-05-12T12:00:57.141 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 39000000, 2021-05-12T12:00:57.141 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 39000000 2021-05-12T12:00:57.141 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.142 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:00:57.142 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:00:57.142 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.142 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.142 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_stats": [ 2021-05-12T12:00:57.143 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.143 INFO:tasks.workunit.client.0.smithi191.stdout: "poolid": 1, 2021-05-12T12:00:57.143 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:00:57.143 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:00:57.143 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48300, 2021-05-12T12:00:57.144 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:00:57.144 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:00:57.144 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:00:57.144 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:00:57.144 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:00:57.144 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:00:57.145 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:00:57.145 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:00:57.145 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:00:57.145 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:00:57.145 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:00:57.146 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:00:57.146 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 100, 2021-05-12T12:00:57.146 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 100, 2021-05-12T12:00:57.146 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:00:57.146 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:00:57.146 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:00:57.147 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:00:57.147 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:00:57.147 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:00:57.147 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:00:57.147 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:00:57.148 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:00:57.148 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:00:57.148 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:00:57.148 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:00:57.148 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:00:57.148 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:00:57.149 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:00:57.149 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:00:57.149 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:00:57.149 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:00:57.149 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:00:57.150 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:00:57.150 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:00:57.150 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:00:57.150 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:00:57.150 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:00:57.151 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:00:57.151 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.151 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:00:57.151 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:00:57.151 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:00:57.151 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:00:57.152 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 1228800, 2021-05-12T12:00:57.152 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 144900, 2021-05-12T12:00:57.152 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:00:57.152 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:00:57.152 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:00:57.153 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:00:57.153 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:00:57.153 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.153 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 16, 2021-05-12T12:00:57.153 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 16, 2021-05-12T12:00:57.153 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 2, 2021-05-12T12:00:57.154 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 2, 2021-05-12T12:00:57.154 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 3 2021-05-12T12:00:57.154 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.154 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.154 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_stats": [ 2021-05-12T12:00:57.155 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.155 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:00:57.155 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 154618822667 2021-05-12T12:00:57.155 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.155 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.155 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:00:57.156 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 42949672977 2021-05-12T12:00:57.156 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.156 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.156 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:00:57.156 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 163208757258 2021-05-12T12:00:57.157 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.157 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.157 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_state": [ 2021-05-12T12:00:57.157 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.157 INFO:tasks.workunit.client.0.smithi191.stdout: "state": "active+undersized+degraded", 2021-05-12T12:00:57.157 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 1 2021-05-12T12:00:57.158 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.158 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.158 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_osd": [ 2021-05-12T12:00:57.158 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.158 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:00:57.159 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:00:57.159 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:00:57.159 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:00:57.159 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.160 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.160 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:00:57.160 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:00:57.160 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 0, 2021-05-12T12:00:57.160 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:00:57.161 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.161 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.161 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:00:57.161 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 1, 2021-05-12T12:00:57.161 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:00:57.162 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:00:57.162 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.162 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.162 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": [ 2021-05-12T12:00:57.162 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:00:57.163 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:00:57.163 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": {} 2021-05-12T12:00:57.163 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.163 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:00:57.163 INFO:tasks.workunit.client.0.smithi191.stdout: "servicemap": { 2021-05-12T12:00:57.164 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 4, 2021-05-12T12:00:57.164 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:00:13.416972+0000", 2021-05-12T12:00:57.164 INFO:tasks.workunit.client.0.smithi191.stdout: "services": { 2021-05-12T12:00:57.164 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": { 2021-05-12T12:00:57.165 INFO:tasks.workunit.client.0.smithi191.stdout: "daemons": { 2021-05-12T12:00:57.165 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": "", 2021-05-12T12:00:57.165 INFO:tasks.workunit.client.0.smithi191.stdout: "2": { 2021-05-12T12:00:57.165 INFO:tasks.workunit.client.0.smithi191.stdout: "start_epoch": 0, 2021-05-12T12:00:57.165 INFO:tasks.workunit.client.0.smithi191.stdout: "start_stamp": "0.000000", 2021-05-12T12:00:57.165 INFO:tasks.workunit.client.0.smithi191.stdout: "gid": 0, 2021-05-12T12:00:57.166 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:00:57.166 INFO:tasks.workunit.client.0.smithi191.stdout: "metadata": {}, 2021-05-12T12:00:57.166 INFO:tasks.workunit.client.0.smithi191.stdout: "task_status": {} 2021-05-12T12:00:57.166 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.167 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.167 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.167 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.167 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:00:57.167 INFO:tasks.workunit.client.0.smithi191.stdout: "paxos": { 2021-05-12T12:00:57.168 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:00:57.168 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 384, 2021-05-12T12:00:57.168 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pn": 0, 2021-05-12T12:00:57.168 INFO:tasks.workunit.client.0.smithi191.stdout: "accepted_pn": 0 2021-05-12T12:00:57.168 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:00:57.168 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:00:57.170 INFO:tasks.workunit.client.0.smithi191.stderr:report 3909156982 2021-05-12T12:00:57.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:00:57.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2021-05-12T12:00:57.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2021-05-12T12:00:57.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2021-05-12T12:00:57.171 INFO:tasks.workunit.client.0.smithi191.stdout:writing non-divergent object existing_4 2021-05-12T12:00:57.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2021-05-12T12:00:57.280 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:00:57.280 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:00:21.887662+0000 29'100 43:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:00:57.281 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:00:57.281 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:00:57.281 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:00:57.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T12:00:57.412 INFO:tasks.workunit.client.0.smithi191.stdout:delay recovery 2021-05-12T12:00:57.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2021-05-12T12:00:57.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2021-05-12T12:00:57.837 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:00:57.837 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:00:21.887662+0000 29'100 43:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:00:57.837 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:00:57.837 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:00:57.839 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:00:57.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:57.849 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2021-05-12T12:00:57.850 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:00:57.850 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:00:57.850 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:00:57.850 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:57.851 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:00:57.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.0.asok 2021-05-12T12:00:57.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-05-12T12:00:57.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T12:00:57.944 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:00:57.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:57.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2021-05-12T12:00:57.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:00:57.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:00:57.954 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:00:57.955 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:57.955 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:00:57.955 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.2.asok 2021-05-12T12:00:57.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-05-12T12:00:57.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T12:00:58.049 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:00:58.057 INFO:tasks.workunit.client.0.smithi191.stdout:revive divergent 1 2021-05-12T12:00:58.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2021-05-12T12:00:58.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2021-05-12T12:00:58.397 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:00:58.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:00:21.887662+0000 29'100 43:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:00:58.398 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:00:58.398 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:00:58.399 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:00:58.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2021-05-12T12:00:59.133 INFO:tasks.workunit.client.0.smithi191.stderr:noup is set 2021-05-12T12:00:59.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2021-05-12T12:00:59.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:00:59.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:00:59.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:00:59.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:00:59.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:00:59.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:00:59.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:00:59.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:00:59.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:00:59.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:00:59.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:00:59.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:00:59.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:00:59.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:00:59.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:00:59.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:00:59.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:00:59.150 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:00:59.150 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:59.150 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:00:59.150 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:00:59.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:00:59.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:00:59.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:00:59.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:00:59.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:00:59.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:00:59.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:59.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:00:59.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:00:59.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:00:59.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:00:59.153 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:00:59.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:59.155 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:00:59.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:00:59.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:00:59.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:00:59.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:00:59.172 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:59.170+0000 7f859c6930c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:59.188 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:59.186+0000 7f859c6930c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:59.190 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:59.187+0000 7f859c6930c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:59.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T12:00:59.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2021-05-12T12:00:59.706 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:00:59.704+0000 7f859c6930c0 -1 Falling back to public interface 2021-05-12T12:01:02.529 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:02.526+0000 7f859c6930c0 -1 osd.1 29 log_to_monitors {default=true} 2021-05-12T12:01:04.514 INFO:tasks.workunit.client.0.smithi191.stdout:delay recovery divergent 2021-05-12T12:01:04.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2021-05-12T12:01:04.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:04.867 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:04.867 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:00:21.887662+0000 43'101 43:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:01:04.868 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:01:04.868 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:04.869 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:01:04.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2021-05-12T12:01:04.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:01:04.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:01:04.880 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:01:04.880 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:04.880 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:04.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.1.asok 2021-05-12T12:01:04.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2021-05-12T12:01:04.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T12:01:04.974 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:01:04.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2021-05-12T12:01:05.685 INFO:tasks.workunit.client.0.smithi191.stderr:noup is unset 2021-05-12T12:01:05.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2021-05-12T12:01:05.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:05.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:01:05.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:05.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:05.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:05.696 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:01:05.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:05.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:05.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:01:06.081 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/42157,v1:127.0.0.1:6803/42157] [v2:127.0.0.1:6804/42157,v1:127.0.0.1:6805/42157] exists,up 076f4795-1efd-4143-b86e-b70ffb6f20f8 2021-05-12T12:01:06.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:01:06.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:01:06.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:01:06.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2021-05-12T12:01:06.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:06.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:01:06.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:06.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:06.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:06.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:06.085 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:01:06.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:06.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:01:06.443 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/44480,v1:127.0.0.1:6819/44480] [v2:127.0.0.1:6820/44480,v1:127.0.0.1:6821/44480] exists,up 1f571db8-2fee-4731-aa84-e47ff0b8961f 2021-05-12T12:01:06.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:01:06.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:01:06.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:01:06.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2021-05-12T12:01:06.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:06.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:01:06.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:06.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:06.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:06.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:06.447 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:01:06.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:06.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:01:06.809 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/42740,v1:127.0.0.1:6811/42740] [v2:127.0.0.1:6812/42740,v1:127.0.0.1:6813/42740] exists,up e783e361-c2f1-4c7c-946d-95632561588b 2021-05-12T12:01:06.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:01:06.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:01:06.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:01:06.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:07.156 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:07.157 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:00:21.887662+0000 43'101 45:129 [0,2] 0 [0,2] 0 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:01:07.157 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:01:07.157 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:07.159 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:01:07.168 INFO:tasks.workunit.client.0.smithi191.stdout:wait for peering 2021-05-12T12:01:07.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2021-05-12T12:01:07.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:07.511 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:07.511 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:00:21.887662+0000 43'101 45:129 [0,2] 0 [0,2] 0 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:01:07.512 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:01:07.512 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:07.513 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:01:07.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2021-05-12T12:01:18.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2021-05-12T12:01:18.012 INFO:tasks.workunit.client.0.smithi191.stdout:killing divergent 1 2021-05-12T12:01:18.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:18.366 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:18.367 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2021-05-12T12:01:07.060272+0000 43'101 49:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:01:18.367 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:01:18.367 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:18.368 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:01:18.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2021-05-12T12:01:18.379 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:01:18.379 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:01:18.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:01:18.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:01:18.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:01:18.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:01:18.484 INFO:tasks.workunit.client.0.smithi191.stdout:reviving divergent 1 2021-05-12T12:01:18.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2021-05-12T12:01:18.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:18.826 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:18.826 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2021-05-12T12:01:07.060272+0000 43'101 49:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:01:18.827 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:01:18.827 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:18.828 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:01:18.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2021-05-12T12:01:18.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:01:18.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:01:18.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:01:18.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:01:18.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:01:18.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:01:18.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:01:18.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:01:18.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:01:18.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:01:18.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:01:18.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:01:18.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:01:18.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:01:18.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:01:18.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:01:18.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:01:18.845 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:01:18.845 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:18.845 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:18.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:01:18.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:01:18.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:01:18.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:01:18.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:01:18.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:01:18.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:01:18.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:18.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:01:18.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:01:18.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:01:18.848 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:01:18.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:01:18.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:18.850 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:01:18.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:01:18.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:01:18.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:01:18.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:01:18.868 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:18.865+0000 7f691e6590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:18.869 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:18.867+0000 7f691e6590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:18.870 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:18.868+0000 7f691e6590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:19.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:01:19.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:19.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:01:19.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:19.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:19.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:19.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:19.237 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:01:19.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:19.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:01:19.385 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:19.383+0000 7f691e6590c0 -1 Falling back to public interface 2021-05-12T12:01:19.618 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/44480,v1:127.0.0.1:6819/44480] [v2:127.0.0.1:6820/44480,v1:127.0.0.1:6821/44480] exists,up 1f571db8-2fee-4731-aa84-e47ff0b8961f 2021-05-12T12:01:19.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:01:19.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:01:19.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:01:19.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2021-05-12T12:01:21.920 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:21.917+0000 7f691e6590c0 -1 osd.1 49 log_to_monitors {default=true} 2021-05-12T12:01:39.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2021-05-12T12:01:39.621 INFO:tasks.workunit.client.0.smithi191.stdout:allowing recovery 2021-05-12T12:01:39.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:39.972 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:39.972 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2021-05-12T12:01:27.017961+0000 49'102 54:152 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:01:39.972 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:01:39.973 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:39.974 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:01:39.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-05-12T12:01:39.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T12:01:40.107 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:01:40.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-05-12T12:01:40.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T12:01:40.240 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:01:40.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-05-12T12:01:40.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T12:01:40.373 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:01:40.382 INFO:tasks.workunit.client.0.smithi191.stdout:reading divergent objects 2021-05-12T12:01:40.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2021-05-12T12:01:40.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:40.723 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:40.723 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2021-05-12T12:01:27.017961+0000 49'102 54:152 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T11:59:25.447276+0000 0'0 2021-05-12T11:59:25.447276+0000 0 2021-05-12T12:01:40.723 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:01:40.723 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:40.724 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:01:40.734 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2021-05-12T12:01:40.735 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2021-05-12T12:01:40.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:01:40.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T12:01:40.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:01:40.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T12:01:40.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:01:40.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T12:01:40.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:01:40.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T12:01:40.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2021-05-12T12:01:40.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-05-12T12:01:40.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:01:40.871 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:06.859+0000 7f8578589700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4536.0:1 2021-05-12T11:59:56.021358+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:01:40.871 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:06.859+0000 7f8578589700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4536.0:1 2021-05-12T11:59:56.021358+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:01:40.871 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:06.859+0000 7f8578589700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:01:40.872 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:06.859+0000 7f8578589700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2021-05-12T12:01:40.872 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:06.859+0000 7f8578589700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2021-05-12T12:01:40.872 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:06.859+0000 7f8578589700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:01:40.872 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:06.859+0000 7f8578589700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2021-05-12T12:01:40.873 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:06.859+0000 7f8578589700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4536.0:1 2021-05-12T11:59:56.021358+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:01:40.873 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:06.859+0000 7f8578589700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-05-12T12:01:40.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-05-12T12:01:40.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:01:40.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2021-05-12T12:01:40.875 INFO:tasks.workunit.client.0.smithi191.stdout:success 2021-05-12T12:01:40.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2021-05-12T12:01:40.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T12:01:40.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T12:01:41.360 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test' does not exist 2021-05-12T12:01:41.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2021-05-12T12:01:41.371 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:01:41.371 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:01:41.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:01:41.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:01:41.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:01:41.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:01:41.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T12:01:41.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:01:41.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:01:41.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:01:41.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:01:41.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:01:41.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:01:41.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:01:41.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:01:41.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:01:41.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:01:41.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:01:41.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:01:41.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:01:41.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:01:41.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:01:41.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:01:41.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:01:41.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:01:41.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:01:41.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:01:41.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:01:41.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:01:41.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:01:41.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:41.500 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:41.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34183 2021-05-12T12:01:41.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:01:41.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:01:41.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T12:01:41.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T12:01:41.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T12:01:41.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T12:01:41.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:01:41.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:01:41.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:01:41.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:01:41.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:01:41.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:01:41.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:01:41.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:01:41.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:01:41.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:01:41.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:01:41.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:01:41.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:01:41.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:01:41.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:01:41.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:01:41.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:01:41.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:01:41.517 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:01:41.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:01:41.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:01:41.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:01:41.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:01:41.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:41.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:41.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34183 2021-05-12T12:01:41.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:01:41.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:01:41.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T12:01:41.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:01:41.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:41.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:41.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34183 2021-05-12T12:01:41.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:01:41.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:01:41.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:01:41.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T12:01:41.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2021-05-12T12:01:41.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2021-05-12T12:01:41.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2021-05-12T12:01:41.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2021-05-12T12:01:41.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2021-05-12T12:01:41.522 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2021-05-12T12:01:41.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2021-05-12T12:01:41.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2021-05-12T12:01:41.523 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:01:41.523 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:01:41.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2021-05-12T12:01:41.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T12:01:41.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:01:41.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:01:41.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:01:41.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T12:01:41.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T12:01:41.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:01:41.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:01:41.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:01:41.906 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:01:41.906 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:41.907 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:41.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:01:41.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:01:42.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:01:42.160 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:01:42.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:01:42.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:01:42.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:01:42.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:01:42.162 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:01:42.163 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:01:42.163 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:01:42.164 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:01:42.164 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:42.165 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:42.165 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-mon.a.asok 2021-05-12T12:01:42.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:01:42.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.34183/ceph-mon.a.asok config get fsid 2021-05-12T12:01:42.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:01:42.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:01:42.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:01:42.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:01:42.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:01:42.255 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:01:42.255 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:01:42.255 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:01:42.255 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:01:42.256 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:42.256 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:42.256 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-mon.a.asok 2021-05-12T12:01:42.256 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:01:42.256 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.34183/ceph-mon.a.asok config get mon_host 2021-05-12T12:01:42.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2021-05-12T12:01:42.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T12:01:42.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:01:42.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:01:42.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:01:42.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T12:01:42.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:01:42.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:01:42.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:01:42.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:01:42.844 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:01:42.845 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:42.845 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:42.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:01:42.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:01:42.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:01:42.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-05-12T12:01:42.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2021-05-12T12:01:42.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:01:42.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:01:42.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:01:42.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:01:42.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:01:42.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:01:42.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:01:42.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:01:42.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:01:42.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:01:42.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:01:42.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:01:42.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:01:42.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:01:42.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:01:42.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:01:42.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:01:42.879 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:01:42.880 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:42.880 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:42.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:01:42.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:01:42.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:01:42.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:01:42.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:01:42.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:01:42.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:01:42.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:01:42.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:01:42.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:42.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:01:42.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:01:42.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:01:42.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:01:42.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=680f25cc-1994-4806-b5f0-2bb987fe27cd 2021-05-12T12:01:42.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 680f25cc-1994-4806-b5f0-2bb987fe27cd' 2021-05-12T12:01:42.884 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 680f25cc-1994-4806-b5f0-2bb987fe27cd 2021-05-12T12:01:42.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:01:42.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCmw5tgAc44NRAALkLpe8TMQD1XIiRcnKvTIA== 2021-05-12T12:01:42.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCmw5tgAc44NRAALkLpe8TMQD1XIiRcnKvTIA=="}' 2021-05-12T12:01:42.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 680f25cc-1994-4806-b5f0-2bb987fe27cd -i td/divergent-priors/0/new.json 2021-05-12T12:01:43.215 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:01:43.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T12:01:43.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCmw5tgAc44NRAALkLpe8TMQD1XIiRcnKvTIA== --osd-uuid 680f25cc-1994-4806-b5f0-2bb987fe27cd 2021-05-12T12:01:43.257 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:43.255+0000 7f3416c890c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:43.259 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:43.257+0000 7f3416c890c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:43.260 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:43.257+0000 7f3416c890c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:43.261 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:43.258+0000 7f3416c890c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:01:43.261 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:43.258+0000 7f3416c890c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:01:43.261 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:43.258+0000 7f3416c890c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:01:43.306 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:43.304+0000 7f3416c890c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T12:01:47.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T12:01:47.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:01:47.988 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:01:47.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:01:47.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:01:48.476 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:01:48.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:01:48.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:48.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:01:48.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:01:48.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:01:48.505 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:48.503+0000 7f679246b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:48.514 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:48.511+0000 7f679246b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:48.515 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:48.513+0000 7f679246b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:48.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:01:48.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:48.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:01:48.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:48.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:48.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:48.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:48.849 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:01:48.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:48.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:01:49.029 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:49.026+0000 7f679246b0c0 -1 Falling back to public interface 2021-05-12T12:01:49.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:01:50.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:01:50.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:50.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:01:50.228 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:01:50.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:50.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:01:50.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:01:51.267 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:51.264+0000 7f679246b0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:01:51.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:01:51.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:51.596 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:01:51.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:01:51.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:51.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:01:51.972 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/47126,v1:127.0.0.1:6803/47126] [v2:127.0.0.1:6804/47126,v1:127.0.0.1:6805/47126] exists,up 680f25cc-1994-4806-b5f0-2bb987fe27cd 2021-05-12T12:01:51.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:01:51.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:01:51.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:01:51.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-05-12T12:01:51.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2021-05-12T12:01:51.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:01:51.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:01:51.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:01:51.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:01:51.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:01:51.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:01:51.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:01:51.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:01:51.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:01:51.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:01:51.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:01:51.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:01:51.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:01:51.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:01:51.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:01:51.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:01:51.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:01:51.981 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:01:51.981 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:51.981 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:01:51.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:01:51.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:01:51.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:01:51.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:01:51.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:01:51.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:01:51.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:01:51.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:01:51.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:01:51.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:51.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:01:51.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:01:51.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:01:51.984 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:01:51.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a 2021-05-12T12:01:51.985 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a 2021-05-12T12:01:51.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a' 2021-05-12T12:01:51.986 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:01:51.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCvw5tgN2c6OxAAL2buxbQqv52YXj2S5dcuXQ== 2021-05-12T12:01:51.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCvw5tgN2c6OxAAL2buxbQqv52YXj2S5dcuXQ=="}' 2021-05-12T12:01:51.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a -i td/divergent-priors/1/new.json 2021-05-12T12:01:52.555 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:01:52.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T12:01:52.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCvw5tgN2c6OxAAL2buxbQqv52YXj2S5dcuXQ== --osd-uuid 8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a 2021-05-12T12:01:52.597 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:52.594+0000 7f846e91b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:52.613 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:52.611+0000 7f846e91b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:52.614 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:52.611+0000 7f846e91b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:52.615 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:52.612+0000 7f846e91b0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:01:52.615 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:52.612+0000 7f846e91b0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:01:52.615 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:52.612+0000 7f846e91b0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:01:52.729 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:52.727+0000 7f846e91b0c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T12:01:57.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T12:01:57.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:01:57.573 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:01:57.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:01:57.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:01:58.066 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:01:58.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:01:58.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:01:58.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:58.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:01:58.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:01:58.094 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:58.091+0000 7f556eef10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:58.097 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:58.094+0000 7f556eef10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:58.099 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:58.097+0000 7f556eef10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:58.434 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:01:58.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:58.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:01:58.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:58.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:58.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:58.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:58.436 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:01:58.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:58.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:01:58.614 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:01:58.612+0000 7f556eef10c0 -1 Falling back to public interface 2021-05-12T12:01:58.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:01:59.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:01:59.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:59.812 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:01:59.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:01:59.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:59.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:02:00.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:00.884 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:00.881+0000 7f556eef10c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:02:01.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:01.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:01.183 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:02:01.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:02:01.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:01.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:02:01.615 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/48511,v1:127.0.0.1:6811/48511] [v2:127.0.0.1:6812/48511,v1:127.0.0.1:6813/48511] exists,up 8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a 2021-05-12T12:02:01.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:02:01.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:02:01.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:02:01.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-05-12T12:02:01.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2021-05-12T12:02:01.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:02:01.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:02:01.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:02:01.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:02:01.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:02:01.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:02:01.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:02:01.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:02:01.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:02:01.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:02:01.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:02:01.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:02:01.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:02:01.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:02:01.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:02:01.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:02:01.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:02:01.624 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:02:01.624 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:01.624 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:02:01.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:02:01.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:02:01.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:02:01.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:02:01.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:02:01.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:02:01.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:02:01.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:02:01.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:02:01.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:01.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:02:01.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:02:01.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:02:01.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:02:01.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=9dbbea6e-ca5b-4917-9e87-b424e890b08e 2021-05-12T12:02:01.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 9dbbea6e-ca5b-4917-9e87-b424e890b08e' 2021-05-12T12:02:01.629 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 9dbbea6e-ca5b-4917-9e87-b424e890b08e 2021-05-12T12:02:01.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:02:01.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC5w5tg6DnbJRAAKolwy+FEVw2SQz4MEtYikA== 2021-05-12T12:02:01.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC5w5tg6DnbJRAAKolwy+FEVw2SQz4MEtYikA=="}' 2021-05-12T12:02:01.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 9dbbea6e-ca5b-4917-9e87-b424e890b08e -i td/divergent-priors/2/new.json 2021-05-12T12:02:02.121 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:02:02.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T12:02:02.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC5w5tg6DnbJRAAKolwy+FEVw2SQz4MEtYikA== --osd-uuid 9dbbea6e-ca5b-4917-9e87-b424e890b08e 2021-05-12T12:02:02.162 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:02.158+0000 7f7bdb21b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:02.163 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:02.160+0000 7f7bdb21b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:02.166 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:02.161+0000 7f7bdb21b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:02.166 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:02.162+0000 7f7bdb21b0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:02:02.167 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:02.162+0000 7f7bdb21b0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:02:02.167 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:02.162+0000 7f7bdb21b0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:02:02.247 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:02.244+0000 7f7bdb21b0c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T12:02:07.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T12:02:07.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:02:07.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:02:07.146 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:02:07.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:02:07.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:02:07.685 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:02:07.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:02:07.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:07.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:02:07.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:02:07.712 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:07.709+0000 7f4f960190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:07.769 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:07.766+0000 7f4f960190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:07.770 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:07.768+0000 7f4f960190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:08.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:02:08.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:02:08.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:02:08.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:02:08.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:02:08.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:08.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:02:08.120 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:02:08.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:08.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:02:08.285 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:08.282+0000 7f4f960190c0 -1 Falling back to public interface 2021-05-12T12:02:08.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:09.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:09.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:09.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:02:09.485 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:02:09.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:09.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:02:09.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:10.550 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:10.548+0000 7f4f960190c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:02:10.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:10.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:10.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:02:10.860 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:02:10.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:10.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:02:11.286 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/49898,v1:127.0.0.1:6819/49898] [v2:127.0.0.1:6820/49898,v1:127.0.0.1:6821/49898] exists,up 9dbbea6e-ca5b-4917-9e87-b424e890b08e 2021-05-12T12:02:11.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:02:11.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:02:11.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:02:11.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2021-05-12T12:02:11.976 INFO:tasks.workunit.client.0.smithi191.stderr:noout is set 2021-05-12T12:02:11.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2021-05-12T12:02:12.670 INFO:tasks.workunit.client.0.smithi191.stderr:noin is set 2021-05-12T12:02:12.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2021-05-12T12:02:13.531 INFO:tasks.workunit.client.0.smithi191.stderr:nodown is set 2021-05-12T12:02:13.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2021-05-12T12:02:13.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-05-12T12:02:14.050 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test' already exists 2021-05-12T12:02:14.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:02:15.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2021-05-12T12:02:16.005 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 size to 3 2021-05-12T12:02:16.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2021-05-12T12:02:16.715 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 min_size to 2 2021-05-12T12:02:16.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2021-05-12T12:02:16.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:02:16.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:02:17.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:02:17.088 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:02:17.089 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:02:17.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:02:17.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:17.089 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:02:17.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836488 2021-05-12T12:02:17.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836488 2021-05-12T12:02:17.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488' 2021-05-12T12:02:17.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:17.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:02:17.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672966 2021-05-12T12:02:17.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672966 2021-05-12T12:02:17.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2021-05-12T12:02:17.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:17.378 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:02:17.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542148 2021-05-12T12:02:17.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542148 2021-05-12T12:02:17.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-60129542148' 2021-05-12T12:02:17.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:17.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2021-05-12T12:02:17.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:17.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:02:17.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836488 2021-05-12T12:02:17.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:17.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836488 2021-05-12T12:02:17.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-05-12T12:02:17.516 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836488 2021-05-12T12:02:17.516 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:02:17.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836488 2021-05-12T12:02:17.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:17.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-05-12T12:02:17.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:17.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:02:17.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672966 2021-05-12T12:02:17.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:17.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672966 2021-05-12T12:02:17.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-05-12T12:02:17.883 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672966 2021-05-12T12:02:17.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:02:18.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672966 2021-05-12T12:02:18.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:18.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542148 2021-05-12T12:02:18.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:18.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:02:18.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542148 2021-05-12T12:02:18.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:18.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542148 2021-05-12T12:02:18.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2021-05-12T12:02:18.251 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542148 2021-05-12T12:02:18.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:02:18.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542146 -lt 60129542148 2021-05-12T12:02:18.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:02:19.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:02:19.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:02:20.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542148 -lt 60129542148 2021-05-12T12:02:20.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2021-05-12T12:02:20.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:02:20.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:02:20.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:02:20.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:02:20.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:02:20.008 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:02:20.008 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:02:20.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:02:20.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:02:20.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:02:20.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:02:20.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:02:20.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:02:20.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:02:20.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:02:20.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:02:20.465 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:02:20.465 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:02:20.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:02:20.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:20.466 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:02:20.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836491 2021-05-12T12:02:20.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836491 2021-05-12T12:02:20.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491' 2021-05-12T12:02:20.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:20.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:02:20.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672968 2021-05-12T12:02:20.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672968 2021-05-12T12:02:20.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-42949672968' 2021-05-12T12:02:20.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:20.754 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:02:20.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542150 2021-05-12T12:02:20.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542150 2021-05-12T12:02:20.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-42949672968 2-60129542150' 2021-05-12T12:02:20.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:20.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-05-12T12:02:20.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:20.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:02:20.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836491 2021-05-12T12:02:20.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:20.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836491 2021-05-12T12:02:20.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-05-12T12:02:20.890 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836491 2021-05-12T12:02:20.891 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:02:21.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836489 -lt 21474836491 2021-05-12T12:02:21.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:02:22.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:02:22.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:02:22.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836491 2021-05-12T12:02:22.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:22.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-05-12T12:02:22.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:22.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:02:22.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672968 2021-05-12T12:02:22.631 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:22.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672968 2021-05-12T12:02:22.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-05-12T12:02:22.632 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672968 2021-05-12T12:02:22.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:02:22.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672968 -lt 42949672968 2021-05-12T12:02:22.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:22.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542150 2021-05-12T12:02:22.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:22.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:02:22.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542150 2021-05-12T12:02:22.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:22.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542150 2021-05-12T12:02:23.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2021-05-12T12:02:23.000 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542150 2021-05-12T12:02:23.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:02:23.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542150 -lt 60129542150 2021-05-12T12:02:23.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:02:23.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:02:23.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:02:23.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:02:23.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:02:23.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:02:23.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:02:23.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:02:23.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:02:23.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:02:23.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:02:24.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:02:24.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:02:24.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:02:24.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:02:24.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:02:24.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:02:24.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:02:24.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2021-05-12T12:02:24.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2021-05-12T12:02:25.074 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:02:25.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2021-05-12T12:02:25.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2021-05-12T12:02:25.085 INFO:tasks.workunit.client.0.smithi191.stdout:primary and soon to be divergent is 1 2021-05-12T12:02:25.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:02:25.430 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:02:25.430 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:02:15.439089+0000 0'0 26:13 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:02:25.430 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:02:25.431 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:02:25.432 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:02:25.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2021-05-12T12:02:25.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-05-12T12:02:25.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2021-05-12T12:02:25.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2021-05-12T12:02:25.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-05-12T12:02:25.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2021-05-12T12:02:25.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2021-05-12T12:02:25.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-05-12T12:02:25.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2021-05-12T12:02:25.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2021-05-12T12:02:25.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2021-05-12T12:02:25.445 INFO:tasks.workunit.client.0.smithi191.stdout:writing initial objects 2021-05-12T12:02:25.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2021-05-12T12:02:25.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:25.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2021-05-12T12:02:25.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:25.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2021-05-12T12:02:25.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:25.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2021-05-12T12:02:25.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:25.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2021-05-12T12:02:25.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:25.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2021-05-12T12:02:26.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:26.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2021-05-12T12:02:26.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:26.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2021-05-12T12:02:26.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:26.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2021-05-12T12:02:26.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:26.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2021-05-12T12:02:26.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:26.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2021-05-12T12:02:26.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:26.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2021-05-12T12:02:26.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:26.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2021-05-12T12:02:27.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:27.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2021-05-12T12:02:27.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:27.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2021-05-12T12:02:27.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:27.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2021-05-12T12:02:27.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:27.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2021-05-12T12:02:27.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:27.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2021-05-12T12:02:27.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:27.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2021-05-12T12:02:27.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:27.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2021-05-12T12:02:27.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:27.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2021-05-12T12:02:28.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:28.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2021-05-12T12:02:28.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:28.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2021-05-12T12:02:28.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:28.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2021-05-12T12:02:28.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:28.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2021-05-12T12:02:28.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:28.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2021-05-12T12:02:28.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:28.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2021-05-12T12:02:28.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:28.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2021-05-12T12:02:29.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:29.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2021-05-12T12:02:29.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:29.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2021-05-12T12:02:29.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:29.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2021-05-12T12:02:29.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:29.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2021-05-12T12:02:29.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:29.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2021-05-12T12:02:29.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:29.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2021-05-12T12:02:29.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:29.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2021-05-12T12:02:30.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:30.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2021-05-12T12:02:30.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:30.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2021-05-12T12:02:30.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:30.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2021-05-12T12:02:30.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:30.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2021-05-12T12:02:30.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:30.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2021-05-12T12:02:30.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:30.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2021-05-12T12:02:30.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:30.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2021-05-12T12:02:30.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:30.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2021-05-12T12:02:31.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:31.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2021-05-12T12:02:31.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:31.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2021-05-12T12:02:31.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:31.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2021-05-12T12:02:31.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:31.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2021-05-12T12:02:31.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:31.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2021-05-12T12:02:31.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:31.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2021-05-12T12:02:31.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:31.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2021-05-12T12:02:31.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:31.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2021-05-12T12:02:32.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:32.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2021-05-12T12:02:32.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:32.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2021-05-12T12:02:32.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:32.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2021-05-12T12:02:32.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:32.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2021-05-12T12:02:32.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:32.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2021-05-12T12:02:32.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:32.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2021-05-12T12:02:32.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:32.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2021-05-12T12:02:32.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:32.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2021-05-12T12:02:33.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:33.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2021-05-12T12:02:33.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:33.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2021-05-12T12:02:33.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:33.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2021-05-12T12:02:33.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:33.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2021-05-12T12:02:33.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:33.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2021-05-12T12:02:33.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:33.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2021-05-12T12:02:33.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:33.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2021-05-12T12:02:34.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:34.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2021-05-12T12:02:34.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:34.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2021-05-12T12:02:34.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:34.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2021-05-12T12:02:34.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:34.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2021-05-12T12:02:34.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:34.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2021-05-12T12:02:34.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:34.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2021-05-12T12:02:34.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:34.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2021-05-12T12:02:34.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:34.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2021-05-12T12:02:35.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:35.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2021-05-12T12:02:35.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:35.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2021-05-12T12:02:35.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:35.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2021-05-12T12:02:35.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:35.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2021-05-12T12:02:35.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:35.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2021-05-12T12:02:35.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:35.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2021-05-12T12:02:35.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:35.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2021-05-12T12:02:36.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:36.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2021-05-12T12:02:36.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:36.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2021-05-12T12:02:36.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:36.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2021-05-12T12:02:36.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:36.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2021-05-12T12:02:36.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:36.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2021-05-12T12:02:36.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:36.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2021-05-12T12:02:36.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:36.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2021-05-12T12:02:36.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:36.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2021-05-12T12:02:37.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:37.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2021-05-12T12:02:37.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:37.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2021-05-12T12:02:37.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:37.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2021-05-12T12:02:37.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:37.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2021-05-12T12:02:37.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:37.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2021-05-12T12:02:37.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:37.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2021-05-12T12:02:37.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:37.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2021-05-12T12:02:38.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:38.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2021-05-12T12:02:38.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:38.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2021-05-12T12:02:38.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:38.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2021-05-12T12:02:38.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:38.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2021-05-12T12:02:38.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:02:38.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2021-05-12T12:02:38.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:02:38.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2021-05-12T12:02:38.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:02:38.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:02:38.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:02:38.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:02:38.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:02:38.690 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:02:38.691 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:02:38.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:02:38.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:02:38.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:02:38.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:02:38.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:02:38.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:02:38.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:02:38.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:02:39.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:02:39.115 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:02:39.115 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:02:39.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:02:39.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:39.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:02:39.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836497 2021-05-12T12:02:39.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836497 2021-05-12T12:02:39.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497' 2021-05-12T12:02:39.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:39.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:02:39.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672974 2021-05-12T12:02:39.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672974 2021-05-12T12:02:39.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-42949672974' 2021-05-12T12:02:39.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:39.384 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:02:39.516 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542156 2021-05-12T12:02:39.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542156 2021-05-12T12:02:39.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-42949672974 2-60129542156' 2021-05-12T12:02:39.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:39.517 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836497 2021-05-12T12:02:39.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:39.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:02:39.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836497 2021-05-12T12:02:39.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:39.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836497 2021-05-12T12:02:39.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2021-05-12T12:02:39.520 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836497 2021-05-12T12:02:39.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:02:39.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836497 -lt 21474836497 2021-05-12T12:02:39.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:39.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-05-12T12:02:39.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:39.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:02:39.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672974 2021-05-12T12:02:39.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:39.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672974 2021-05-12T12:02:39.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-05-12T12:02:39.889 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672974 2021-05-12T12:02:39.890 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:02:40.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672974 -lt 42949672974 2021-05-12T12:02:40.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:40.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542156 2021-05-12T12:02:40.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:40.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:02:40.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542156 2021-05-12T12:02:40.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:40.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542156 2021-05-12T12:02:40.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2021-05-12T12:02:40.298 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542156 2021-05-12T12:02:40.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:02:40.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542154 -lt 60129542156 2021-05-12T12:02:40.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:02:41.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:02:41.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:02:42.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542156 -lt 60129542156 2021-05-12T12:02:42.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:02:42.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:02:42.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:02:42.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:02:42.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:02:42.585 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:02:42.586 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:02:42.586 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:02:42.586 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:02:42.586 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:02:42.586 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:02:42.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:02:42.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:02:42.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:02:42.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:02:43.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:02:43.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:02:43.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:02:43.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2021-05-12T12:02:43.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T12:02:43.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T12:02:43.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T12:02:43.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T12:02:43.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2021-05-12T12:02:43.792 INFO:tasks.workunit.client.0.smithi191.stdout:blackholing osds 0 2 2021-05-12T12:02:43.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2021-05-12T12:02:43.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:02:44.136 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:02:44.137 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T12:02:15.439089+0000 27'100 27:113 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:02:44.137 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:02:44.137 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:02:44.139 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:02:44.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:02:44.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2021-05-12T12:02:44.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:02:44.150 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:02:44.150 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:02:44.150 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:44.150 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:02:44.150 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.0.asok 2021-05-12T12:02:44.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:02:44.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T12:02:44.240 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:02:44.240 INFO:tasks.workunit.client.0.smithi191.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:02:44.240 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:02:44.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:02:44.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2021-05-12T12:02:44.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:02:44.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:02:44.250 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:02:44.250 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:44.251 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:02:44.251 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.2.asok 2021-05-12T12:02:44.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:02:44.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.2.asok config set objectstore_blackhole 1 2021-05-12T12:02:44.340 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:02:44.341 INFO:tasks.workunit.client.0.smithi191.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:02:44.341 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:02:44.348 INFO:tasks.workunit.client.0.smithi191.stdout:create new divergent objects 2021-05-12T12:02:44.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2021-05-12T12:02:44.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2021-05-12T12:02:44.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:02:44.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:02:44.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2021-05-12T12:02:44.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2021-05-12T12:02:44.351 INFO:tasks.workunit.client.0.smithi191.stdout:writing divergent objects 2021-05-12T12:02:44.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2021-05-12T12:02:44.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2021-05-12T12:02:44.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T12:02:44.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T12:02:44.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2021-05-12T12:02:44.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2021-05-12T12:02:44.354 INFO:tasks.workunit.client.0.smithi191.stdout:remove divergent objects 2021-05-12T12:02:44.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2021-05-12T12:02:44.356 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2021-05-12T12:02:44.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T12:02:44.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2021-05-12T12:02:44.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2021-05-12T12:02:44.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T12:02:44.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2021-05-12T12:02:44.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2021-05-12T12:02:44.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2021-05-12T12:02:44.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2021-05-12T12:02:44.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2021-05-12T12:02:54.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2021-05-12T12:02:54.362 INFO:tasks.workunit.client.0.smithi191.stderr:rados: no process found 2021-05-12T12:02:54.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2021-05-12T12:02:54.363 INFO:tasks.workunit.client.0.smithi191.stdout:killing all the osds 2021-05-12T12:02:54.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:02:54.714 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:02:54.715 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2021-05-12T12:02:15.439089+0000 27'106 27:119 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:02:54.715 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:02:54.715 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:02:54.716 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:02:54.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2021-05-12T12:02:54.726 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:02:54.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:02:54.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:02:54.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:02:54.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:02:54.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:02:54.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-05-12T12:02:54.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2021-05-12T12:02:55.323 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already down. 2021-05-12T12:02:55.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-05-12T12:02:55.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2021-05-12T12:02:55.984 INFO:tasks.workunit.client.0.smithi191.stderr:osd.1 is already down. 2021-05-12T12:02:55.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-05-12T12:02:55.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2021-05-12T12:02:56.542 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already down. 2021-05-12T12:02:56.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:02:56.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2021-05-12T12:02:57.052 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already out. 2021-05-12T12:02:57.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:02:57.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2021-05-12T12:02:57.570 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already out. 2021-05-12T12:02:57.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2021-05-12T12:02:57.580 INFO:tasks.workunit.client.0.smithi191.stdout:bringing up non_divergent 0 2 2021-05-12T12:02:57.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:02:57.946 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:02:57.947 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2021-05-12T12:02:15.439089+0000 27'106 27:119 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:02:57.947 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:02:57.947 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:02:57.949 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:02:57.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:02:57.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2021-05-12T12:02:57.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:02:57.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:02:57.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:02:57.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:02:57.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:02:57.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:02:57.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:02:57.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:02:57.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:02:57.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:02:57.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:02:57.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:02:57.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:02:57.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:02:57.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:02:57.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:02:57.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:02:57.966 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:02:57.966 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:57.966 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:02:57.966 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:02:57.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:02:57.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:02:57.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:02:57.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:02:57.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:02:57.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:02:57.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:57.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:02:57.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:02:57.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:02:57.969 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:02:57.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:02:57.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:57.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:02:57.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:02:57.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:02:57.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:02:57.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:02:57.989 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:57.986+0000 7fd0e03030c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:57.990 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:57.988+0000 7fd0e03030c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:57.991 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:57.989+0000 7fd0e03030c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:58.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:02:58.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:02:58.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:02:58.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:02:58.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:02:58.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:58.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:02:58.332 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:02:58.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:58.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:02:58.507 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:02:58.505+0000 7fd0e03030c0 -1 Falling back to public interface 2021-05-12T12:02:58.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:59.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:59.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:59.737 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:02:59.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:02:59.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:59.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:03:00.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:03:01.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:03:01.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:01.113 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:03:01.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:03:01.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:01.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:03:01.187 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:03:01.184+0000 7fd0e03030c0 -1 osd.0 27 log_to_monitors {default=true} 2021-05-12T12:03:01.549 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/53991,v1:127.0.0.1:6803/53991] [v2:127.0.0.1:6804/53991,v1:127.0.0.1:6805/53991] exists,up 680f25cc-1994-4806-b5f0-2bb987fe27cd 2021-05-12T12:03:01.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:03:01.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:03:01.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:03:01.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:01.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2021-05-12T12:03:01.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:03:01.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:03:01.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:03:01.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:03:01.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:03:01.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:03:01.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:03:01.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:03:01.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:03:01.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:03:01.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:03:01.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:03:01.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:03:01.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:03:01.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:03:01.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:03:01.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:03:01.559 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:03:01.559 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:03:01.559 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:03:01.559 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:03:01.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:03:01.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:03:01.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:03:01.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:03:01.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:03:01.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:03:01.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:03:01.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:03:01.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:03:01.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:03:01.562 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:03:01.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:03:01.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:03:01.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:03:01.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:03:01.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:03:01.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:03:01.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:03:01.580 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:03:01.577+0000 7f70f51c90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:03:01.581 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:03:01.579+0000 7f70f51c90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:03:01.582 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:03:01.580+0000 7f70f51c90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:03:01.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:03:01.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:03:01.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:03:01.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:03:01.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:03:01.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:01.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:03:01.929 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:03:01.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:01.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:03:02.098 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:03:02.096+0000 7f70f51c90c0 -1 Falling back to public interface 2021-05-12T12:03:02.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:03:03.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:03:03.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:03.297 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:03:03.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:03:03.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:03.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:03:03.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:03:04.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:03:04.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:04.697 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:03:04.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:03:04.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:04.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:03:04.870 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:03:04.868+0000 7f70f51c90c0 -1 osd.2 27 log_to_monitors {default=true} 2021-05-12T12:03:05.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:03:06.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:03:06.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:06.077 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:03:06.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:03:06.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:06.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:03:06.446 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/54548,v1:127.0.0.1:6811/54548] [v2:127.0.0.1:6812/54548,v1:127.0.0.1:6813/54548] exists,up 9dbbea6e-ca5b-4917-9e87-b424e890b08e 2021-05-12T12:03:06.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:03:06.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:03:06.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:03:06.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:06.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2021-05-12T12:03:07.017 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already in. 2021-05-12T12:03:07.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:07.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2021-05-12T12:03:07.624 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already in. 2021-05-12T12:03:07.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:03:07.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2021-05-12T12:03:07.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:03:07.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:03:07.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:03:07.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:03:07.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:03:07.635 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:03:07.636 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:03:07.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:03:07.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:03:07.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:03:07.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:03:07.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:03:07.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:03:07.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:03:07.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:03:08.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:03:08.197 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:03:08.197 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:03:08.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:03:08.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:08.198 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:03:08.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=146028888068 2021-05-12T12:03:08.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 146028888068 2021-05-12T12:03:08.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-146028888068' 2021-05-12T12:03:08.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:08.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:03:08.429 INFO:tasks.workunit.client.0.smithi191.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:03:08.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:03:08.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:03:08.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:03:08.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:08.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:03:08.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=154618822659 2021-05-12T12:03:08.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 154618822659 2021-05-12T12:03:08.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-146028888068 2-154618822659' 2021-05-12T12:03:08.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:03:08.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-146028888068 2021-05-12T12:03:08.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:03:08.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:03:08.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-146028888068 2021-05-12T12:03:08.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:03:08.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=146028888068 2021-05-12T12:03:08.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2021-05-12T12:03:08.569 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 146028888068 2021-05-12T12:03:08.569 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:03:08.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 146028888066 -lt 146028888068 2021-05-12T12:03:08.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:03:09.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:03:09.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:03:10.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 146028888068 -lt 146028888068 2021-05-12T12:03:10.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:03:10.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822659 2021-05-12T12:03:10.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:03:10.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:03:10.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-154618822659 2021-05-12T12:03:10.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:03:10.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=154618822659 2021-05-12T12:03:10.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2021-05-12T12:03:10.308 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 154618822659 2021-05-12T12:03:10.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:03:10.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 154618822659 -lt 154618822659 2021-05-12T12:03:10.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:03:10.716 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:10.716 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:11.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:03:11.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:11.216 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:11.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:11.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:11.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:11.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:11.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:11.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:11.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:11.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:11.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:12.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:12.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:03:12.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:03:12.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:03:12.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:12.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:03:12.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:12.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:12.151 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:12.152 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:12.152 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:12.152 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:12.152 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:12.153 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:12.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:12.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:12.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:12.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:12.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:12.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:12.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:12.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:12.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:12.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:12.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:12.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:12.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:13.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:13.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:13.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:03:13.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:13.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:03:13.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:13.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:13.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:13.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:13.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:13.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:13.665 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:13.665 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:14.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:14.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:14.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:14.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:14.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:14.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:14.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:14.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:14.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:14.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:14.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:14.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:14.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:14.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:14.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:14.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:03:14.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:14.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:03:15.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:15.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:15.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:15.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:15.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:15.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:15.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:15.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:15.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:15.744 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:15.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:15.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:16.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:16.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:16.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:16.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:16.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:16.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:16.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:16.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:16.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:16.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:16.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:16.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:03:16.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:16.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:03:17.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:17.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:17.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:17.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:17.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:17.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:17.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:17.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:17.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:17.984 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:17.984 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:17.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:18.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:18.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:18.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:18.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:18.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:18.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:18.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:18.466 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:18.466 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:18.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:18.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:18.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:03:18.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:18.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:03:20.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:20.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:20.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:20.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:20.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:20.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:20.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:20.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:20.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:20.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:20.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:20.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:21.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:21.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:21.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:21.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:21.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:21.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:21.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:21.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:21.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:21.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:21.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:21.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:03:21.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:21.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:03:25.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:25.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:25.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:25.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:25.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:25.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:25.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:25.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:25.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:25.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:25.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:25.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:25.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:25.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:25.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:25.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:25.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:25.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:25.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:25.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:25.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:26.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:26.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:26.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:03:26.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:26.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:03:32.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:32.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:32.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:32.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:32.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:32.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:32.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:32.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:33.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:33.173 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:33.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:33.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:33.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:33.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:33.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:33.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:33.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:33.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:33.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:33.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:33.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:34.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:34.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:34.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:03:34.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:34.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:03:41.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:41.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:41.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:41.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:41.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:41.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:41.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:41.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:41.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:41.820 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:41.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:41.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:42.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:42.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:42.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:42.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:42.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:42.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:42.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:42.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:42.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:42.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:42.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:42.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:03:42.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:03:43.253 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:03:43.254 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_fingerprint": "bc472bb2-a4e4-4155-b217-c0b688d45a1a", 2021-05-12T12:03:43.254 INFO:tasks.workunit.client.0.smithi191.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:03:43.254 INFO:tasks.workunit.client.0.smithi191.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:03:43.255 INFO:tasks.workunit.client.0.smithi191.stdout: "timestamp": "2021-05-12T12:03:43.250695+0000", 2021-05-12T12:03:43.255 INFO:tasks.workunit.client.0.smithi191.stdout: "tag": "", 2021-05-12T12:03:43.255 INFO:tasks.workunit.client.0.smithi191.stdout: "health": { 2021-05-12T12:03:43.255 INFO:tasks.workunit.client.0.smithi191.stdout: "status": "HEALTH_ERR", 2021-05-12T12:03:43.255 INFO:tasks.workunit.client.0.smithi191.stdout: "checks": { 2021-05-12T12:03:43.256 INFO:tasks.workunit.client.0.smithi191.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:03:43.256 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:03:43.256 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:03:43.256 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:03:43.257 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:03:43.257 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.257 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:03:43.257 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.257 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:03:43.258 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.258 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.258 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:03:43.258 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.259 INFO:tasks.workunit.client.0.smithi191.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:03:43.259 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:03:43.259 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:03:43.259 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:03:43.259 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 17 2021-05-12T12:03:43.260 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.260 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [], 2021-05-12T12:03:43.260 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:03:43.260 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.260 INFO:tasks.workunit.client.0.smithi191.stdout: "OSD_DOWN": { 2021-05-12T12:03:43.261 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:03:43.261 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:03:43.261 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 osds down", 2021-05-12T12:03:43.261 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:03:43.261 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.262 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:03:43.262 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.262 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "osd.1 (root=default,host=smithi191) is down" 2021-05-12T12:03:43.262 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.263 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.263 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:03:43.263 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.263 INFO:tasks.workunit.client.0.smithi191.stdout: "PG_DEGRADED": { 2021-05-12T12:03:43.263 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:03:43.264 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:03:43.264 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2021-05-12T12:03:43.264 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:03:43.264 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.264 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:03:43.265 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.265 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:03:43.265 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.265 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.266 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:03:43.266 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.266 INFO:tasks.workunit.client.0.smithi191.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:03:43.266 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:03:43.266 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:03:43.267 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:03:43.267 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:03:43.267 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.267 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:03:43.267 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.268 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:03:43.268 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.268 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.268 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:03:43.268 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.269 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.269 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:03:43.269 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.269 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.270 INFO:tasks.workunit.client.0.smithi191.stdout: "mutes": [] 2021-05-12T12:03:43.270 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.270 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_first_committed": 1, 2021-05-12T12:03:43.270 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_last_committed": 1, 2021-05-12T12:03:43.270 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap": { 2021-05-12T12:03:43.271 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:03:43.271 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:03:43.271 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:01:41.535760Z", 2021-05-12T12:03:43.271 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:01:41.535760Z", 2021-05-12T12:03:43.271 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release": 17, 2021-05-12T12:03:43.272 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:03:43.272 INFO:tasks.workunit.client.0.smithi191.stdout: "election_strategy": 1, 2021-05-12T12:03:43.272 INFO:tasks.workunit.client.0.smithi191.stdout: "disallowed_leaders: ": "", 2021-05-12T12:03:43.272 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": false, 2021-05-12T12:03:43.273 INFO:tasks.workunit.client.0.smithi191.stdout: "features": { 2021-05-12T12:03:43.273 INFO:tasks.workunit.client.0.smithi191.stdout: "persistent": [ 2021-05-12T12:03:43.273 INFO:tasks.workunit.client.0.smithi191.stdout: "kraken", 2021-05-12T12:03:43.273 INFO:tasks.workunit.client.0.smithi191.stdout: "luminous", 2021-05-12T12:03:43.273 INFO:tasks.workunit.client.0.smithi191.stdout: "mimic", 2021-05-12T12:03:43.274 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap-prune", 2021-05-12T12:03:43.274 INFO:tasks.workunit.client.0.smithi191.stdout: "nautilus", 2021-05-12T12:03:43.274 INFO:tasks.workunit.client.0.smithi191.stdout: "octopus", 2021-05-12T12:03:43.274 INFO:tasks.workunit.client.0.smithi191.stdout: "pacific", 2021-05-12T12:03:43.274 INFO:tasks.workunit.client.0.smithi191.stdout: "elector-pinging", 2021-05-12T12:03:43.275 INFO:tasks.workunit.client.0.smithi191.stdout: "quincy" 2021-05-12T12:03:43.275 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.275 INFO:tasks.workunit.client.0.smithi191.stdout: "optional": [] 2021-05-12T12:03:43.275 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.275 INFO:tasks.workunit.client.0.smithi191.stdout: "mons": [ 2021-05-12T12:03:43.276 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.276 INFO:tasks.workunit.client.0.smithi191.stdout: "rank": 0, 2021-05-12T12:03:43.276 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "a", 2021-05-12T12:03:43.276 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:03:43.277 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.277 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.277 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.277 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:03:43.277 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 0 2021-05-12T12:03:43.278 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.278 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.278 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.278 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:03:43.279 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:03:43.279 INFO:tasks.workunit.client.0.smithi191.stdout: "priority": 0, 2021-05-12T12:03:43.279 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 0, 2021-05-12T12:03:43.279 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_location": "{}" 2021-05-12T12:03:43.279 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.280 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.280 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.280 INFO:tasks.workunit.client.0.smithi191.stdout: "quorum": [ 2021-05-12T12:03:43.280 INFO:tasks.workunit.client.0.smithi191.stdout: 0 2021-05-12T12:03:43.280 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.281 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap": { 2021-05-12T12:03:43.281 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 39, 2021-05-12T12:03:43.281 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:03:43.281 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:01:42.410837+0000", 2021-05-12T12:03:43.282 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:03:07.662094+0000", 2021-05-12T12:03:43.282 INFO:tasks.workunit.client.0.smithi191.stdout: "last_up_change": "2021-05-12T12:03:05.079814+0000", 2021-05-12T12:03:43.282 INFO:tasks.workunit.client.0.smithi191.stdout: "last_in_change": "2021-05-12T12:03:07.343642+0000", 2021-05-12T12:03:43.282 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:03:43.282 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_num": 5800384, 2021-05-12T12:03:43.283 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_set": [ 2021-05-12T12:03:43.283 INFO:tasks.workunit.client.0.smithi191.stdout: "nodown", 2021-05-12T12:03:43.283 INFO:tasks.workunit.client.0.smithi191.stdout: "noin", 2021-05-12T12:03:43.283 INFO:tasks.workunit.client.0.smithi191.stdout: "noout", 2021-05-12T12:03:43.283 INFO:tasks.workunit.client.0.smithi191.stdout: "pglog_hardlimit", 2021-05-12T12:03:43.284 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snapdirs", 2021-05-12T12:03:43.284 INFO:tasks.workunit.client.0.smithi191.stdout: "recovery_deletes", 2021-05-12T12:03:43.284 INFO:tasks.workunit.client.0.smithi191.stdout: "sortbitwise" 2021-05-12T12:03:43.284 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.284 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_version": 9, 2021-05-12T12:03:43.285 INFO:tasks.workunit.client.0.smithi191.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:03:43.285 INFO:tasks.workunit.client.0.smithi191.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:03:43.285 INFO:tasks.workunit.client.0.smithi191.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:03:43.285 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_snapshot": "", 2021-05-12T12:03:43.285 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_max": 1, 2021-05-12T12:03:43.286 INFO:tasks.workunit.client.0.smithi191.stdout: "max_osd": 3, 2021-05-12T12:03:43.286 INFO:tasks.workunit.client.0.smithi191.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:03:43.286 INFO:tasks.workunit.client.0.smithi191.stdout: "min_compat_client": "jewel", 2021-05-12T12:03:43.286 INFO:tasks.workunit.client.0.smithi191.stdout: "require_osd_release": "quincy", 2021-05-12T12:03:43.286 INFO:tasks.workunit.client.0.smithi191.stdout: "pools": [ 2021-05-12T12:03:43.287 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.287 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:03:43.287 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_name": "test", 2021-05-12T12:03:43.287 INFO:tasks.workunit.client.0.smithi191.stdout: "create_time": "2021-05-12T12:02:13.813588+0000", 2021-05-12T12:03:43.288 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": 1, 2021-05-12T12:03:43.288 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_names": "hashpspool", 2021-05-12T12:03:43.288 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:03:43.288 INFO:tasks.workunit.client.0.smithi191.stdout: "size": 3, 2021-05-12T12:03:43.288 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 2, 2021-05-12T12:03:43.289 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_rule": 0, 2021-05-12T12:03:43.289 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:03:43.289 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:03:43.289 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:03:43.289 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:03:43.290 INFO:tasks.workunit.client.0.smithi191.stdout: "object_hash": 2, 2021-05-12T12:03:43.290 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:03:43.290 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num": 1, 2021-05-12T12:03:43.290 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num": 1, 2021-05-12T12:03:43.290 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num_target": 1, 2021-05-12T12:03:43.291 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_target": 1, 2021-05-12T12:03:43.291 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_pending": 1, 2021-05-12T12:03:43.291 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pg_merge_meta": { 2021-05-12T12:03:43.291 INFO:tasks.workunit.client.0.smithi191.stdout: "source_pgid": "0.0", 2021-05-12T12:03:43.291 INFO:tasks.workunit.client.0.smithi191.stdout: "ready_epoch": 0, 2021-05-12T12:03:43.292 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_started": 0, 2021-05-12T12:03:43.292 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": 0, 2021-05-12T12:03:43.292 INFO:tasks.workunit.client.0.smithi191.stdout: "source_version": "0'0", 2021-05-12T12:03:43.293 INFO:tasks.workunit.client.0.smithi191.stdout: "target_version": "0'0" 2021-05-12T12:03:43.293 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.293 INFO:tasks.workunit.client.0.smithi191.stdout: "last_change": "27", 2021-05-12T12:03:43.293 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend": "0", 2021-05-12T12:03:43.294 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:03:43.294 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:03:43.294 INFO:tasks.workunit.client.0.smithi191.stdout: "auid": 0, 2021-05-12T12:03:43.294 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:03:43.294 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_seq": 0, 2021-05-12T12:03:43.295 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_epoch": 0, 2021-05-12T12:03:43.295 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_snaps": [], 2021-05-12T12:03:43.295 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps": "[]", 2021-05-12T12:03:43.295 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_bytes": 0, 2021-05-12T12:03:43.295 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_objects": 0, 2021-05-12T12:03:43.296 INFO:tasks.workunit.client.0.smithi191.stdout: "tiers": [], 2021-05-12T12:03:43.296 INFO:tasks.workunit.client.0.smithi191.stdout: "tier_of": -1, 2021-05-12T12:03:43.296 INFO:tasks.workunit.client.0.smithi191.stdout: "read_tier": -1, 2021-05-12T12:03:43.296 INFO:tasks.workunit.client.0.smithi191.stdout: "write_tier": -1, 2021-05-12T12:03:43.296 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_mode": "none", 2021-05-12T12:03:43.297 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_bytes": 0, 2021-05-12T12:03:43.297 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_objects": 0, 2021-05-12T12:03:43.297 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:03:43.297 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:03:43.297 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:03:43.298 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_flush_age": 0, 2021-05-12T12:03:43.298 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_evict_age": 0, 2021-05-12T12:03:43.298 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profile": "", 2021-05-12T12:03:43.298 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_params": { 2021-05-12T12:03:43.298 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "none" 2021-05-12T12:03:43.299 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.299 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_period": 0, 2021-05-12T12:03:43.299 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_count": 0, 2021-05-12T12:03:43.299 INFO:tasks.workunit.client.0.smithi191.stdout: "use_gmt_hitset": true, 2021-05-12T12:03:43.299 INFO:tasks.workunit.client.0.smithi191.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:03:43.300 INFO:tasks.workunit.client.0.smithi191.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:03:43.300 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:03:43.300 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:03:43.300 INFO:tasks.workunit.client.0.smithi191.stdout: "grade_table": [], 2021-05-12T12:03:43.301 INFO:tasks.workunit.client.0.smithi191.stdout: "stripe_width": 0, 2021-05-12T12:03:43.301 INFO:tasks.workunit.client.0.smithi191.stdout: "expected_num_objects": 0, 2021-05-12T12:03:43.301 INFO:tasks.workunit.client.0.smithi191.stdout: "fast_read": false, 2021-05-12T12:03:43.301 INFO:tasks.workunit.client.0.smithi191.stdout: "options": {}, 2021-05-12T12:03:43.301 INFO:tasks.workunit.client.0.smithi191.stdout: "application_metadata": {} 2021-05-12T12:03:43.302 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.302 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.302 INFO:tasks.workunit.client.0.smithi191.stdout: "osds": [ 2021-05-12T12:03:43.302 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.302 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:03:43.303 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "680f25cc-1994-4806-b5f0-2bb987fe27cd", 2021-05-12T12:03:43.303 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:03:43.303 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:03:43.303 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:03:43.303 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:03:43.304 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 5, 2021-05-12T12:03:43.304 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 27, 2021-05-12T12:03:43.304 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 34, 2021-05-12T12:03:43.304 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 38, 2021-05-12T12:03:43.304 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 28, 2021-05-12T12:03:43.305 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:03:43.305 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:03:43.305 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.305 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.305 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.306 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:03:43.306 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:03:43.306 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.306 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.307 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.307 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:03:43.307 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:03:43.307 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.307 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.308 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.308 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:03:43.308 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.308 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.308 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.309 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:03:43.309 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:03:43.309 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.309 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.309 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.310 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:03:43.310 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:03:43.310 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.310 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.310 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.311 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:03:43.311 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.311 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.311 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.311 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:03:43.312 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:03:43.312 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.312 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.312 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.312 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:03:43.313 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:03:43.313 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.313 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.313 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.314 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:03:43.314 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.314 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.314 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.314 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:03:43.315 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:03:43.315 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.315 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.315 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.315 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:03:43.316 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:03:43.316 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.316 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.316 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.316 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6803/53991", 2021-05-12T12:03:43.317 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6805/53991", 2021-05-12T12:03:43.317 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6809/53991", 2021-05-12T12:03:43.317 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6807/53991", 2021-05-12T12:03:43.317 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:03:43.317 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:03:43.318 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:03:43.318 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.318 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.318 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.318 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:03:43.319 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a", 2021-05-12T12:03:43.319 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 0, 2021-05-12T12:03:43.319 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:03:43.319 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:03:43.319 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:03:43.320 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 0, 2021-05-12T12:03:43.320 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 0, 2021-05-12T12:03:43.320 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 10, 2021-05-12T12:03:43.320 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 21, 2021-05-12T12:03:43.320 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 29, 2021-05-12T12:03:43.321 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:03:43.321 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:03:43.321 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.321 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.322 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.322 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:03:43.322 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:03:43.322 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.322 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.323 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.323 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:03:43.323 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:03:43.323 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.323 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.324 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.324 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:03:43.324 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.324 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.324 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.325 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:03:43.325 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:03:43.325 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.325 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.325 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.326 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:03:43.326 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:03:43.326 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.326 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.326 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.327 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:03:43.327 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.327 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.327 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.327 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:03:43.328 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:03:43.328 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.328 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.328 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.328 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:03:43.329 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:03:43.329 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.329 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.329 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.330 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:03:43.330 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.330 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.330 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.330 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:03:43.331 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:03:43.331 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.331 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.331 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.331 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:03:43.332 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:03:43.332 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.332 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.332 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.332 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/48511", 2021-05-12T12:03:43.333 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/48511", 2021-05-12T12:03:43.333 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48511", 2021-05-12T12:03:43.333 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48511", 2021-05-12T12:03:43.333 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:03:43.333 INFO:tasks.workunit.client.0.smithi191.stdout: "exists" 2021-05-12T12:03:43.334 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.334 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.334 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.334 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:03:43.334 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "9dbbea6e-ca5b-4917-9e87-b424e890b08e", 2021-05-12T12:03:43.335 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:03:43.335 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:03:43.335 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:03:43.335 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:03:43.336 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 14, 2021-05-12T12:03:43.336 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 27, 2021-05-12T12:03:43.336 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 36, 2021-05-12T12:03:43.336 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 0, 2021-05-12T12:03:43.336 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 30, 2021-05-12T12:03:43.337 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:03:43.337 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:03:43.337 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.337 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.337 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.338 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:03:43.338 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:03:43.338 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.338 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.338 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.339 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:03:43.339 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:03:43.339 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.339 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.339 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.340 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:03:43.340 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.340 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.340 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.340 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:03:43.341 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:03:43.341 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.341 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.341 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.341 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:03:43.342 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:03:43.342 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.342 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.342 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.343 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:03:43.343 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.343 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.343 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.343 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:03:43.344 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:03:43.344 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.344 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.344 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.344 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:03:43.345 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:03:43.345 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.345 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.345 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.346 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:03:43.346 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:03:43.346 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.346 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:03:43.346 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:03:43.347 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:03:43.347 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.347 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.347 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:03:43.347 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:03:43.348 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:03:43.348 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.348 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.348 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.349 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/54548", 2021-05-12T12:03:43.349 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/54548", 2021-05-12T12:03:43.349 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/54548", 2021-05-12T12:03:43.349 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/54548", 2021-05-12T12:03:43.349 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:03:43.350 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:03:43.350 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:03:43.350 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.350 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.350 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.351 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_xinfo": [ 2021-05-12T12:03:43.351 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.351 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:03:43.351 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:02:55.145006+0000", 2021-05-12T12:03:43.351 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:03:43.352 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:03:43.352 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:03:43.352 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:03:43.352 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:01:52.281020+0000", 2021-05-12T12:03:43.353 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:03:43.353 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.353 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.353 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:03:43.353 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:02:55.701124+0000", 2021-05-12T12:03:43.354 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:03:43.354 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:03:43.354 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:03:43.354 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:03:43.354 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:01.884752+0000", 2021-05-12T12:03:43.355 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:03:43.355 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.355 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.355 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:03:43.355 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:02:56.304965+0000", 2021-05-12T12:03:43.356 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:03:43.356 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:03:43.356 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:03:43.356 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:03:43.356 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:11.506727+0000", 2021-05-12T12:03:43.357 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:03:43.357 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.357 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.357 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap": [], 2021-05-12T12:03:43.357 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap_items": [], 2021-05-12T12:03:43.358 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_temp": [], 2021-05-12T12:03:43.358 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_temp": [], 2021-05-12T12:03:43.358 INFO:tasks.workunit.client.0.smithi191.stdout: "blocklist": {}, 2021-05-12T12:03:43.358 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profiles": { 2021-05-12T12:03:43.359 INFO:tasks.workunit.client.0.smithi191.stdout: "default": { 2021-05-12T12:03:43.359 INFO:tasks.workunit.client.0.smithi191.stdout: "k": "2", 2021-05-12T12:03:43.359 INFO:tasks.workunit.client.0.smithi191.stdout: "m": "2", 2021-05-12T12:03:43.359 INFO:tasks.workunit.client.0.smithi191.stdout: "plugin": "jerasure", 2021-05-12T12:03:43.359 INFO:tasks.workunit.client.0.smithi191.stdout: "technique": "reed_sol_van" 2021-05-12T12:03:43.360 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.360 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.360 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps_queue": [], 2021-05-12T12:03:43.360 INFO:tasks.workunit.client.0.smithi191.stdout: "new_removed_snaps": [], 2021-05-12T12:03:43.360 INFO:tasks.workunit.client.0.smithi191.stdout: "new_purged_snaps": [], 2021-05-12T12:03:43.361 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_node_flags": {}, 2021-05-12T12:03:43.361 INFO:tasks.workunit.client.0.smithi191.stdout: "device_class_flags": {}, 2021-05-12T12:03:43.361 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": { 2021-05-12T12:03:43.361 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_enabled": false, 2021-05-12T12:03:43.361 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_bucket_count": 0, 2021-05-12T12:03:43.362 INFO:tasks.workunit.client.0.smithi191.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:03:43.362 INFO:tasks.workunit.client.0.smithi191.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:03:43.362 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_bucket": 0 2021-05-12T12:03:43.362 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.362 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.363 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_metadata": [ 2021-05-12T12:03:43.363 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.363 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:03:43.363 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:03:43.363 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6804/53991,v1:127.0.0.1:6805/53991]", 2021-05-12T12:03:43.364 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:03:43.364 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:03:43.364 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:03:43.364 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:03:43.364 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:03:43.365 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:03:43.365 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:03:43.365 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:03:43.365 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:03:43.365 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:03:43.366 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:03:43.366 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:03:43.366 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:03:43.366 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:03:43.366 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:03:43.367 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:03:43.367 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:03:43.367 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:03:43.367 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:03:43.368 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:03:43.368 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:03:43.368 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:03:43.368 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:03:43.368 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:03:43.369 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:03:43.369 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6802/53991,v1:127.0.0.1:6803/53991]", 2021-05-12T12:03:43.369 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:03:43.369 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/53991,v1:127.0.0.1:6809/53991]", 2021-05-12T12:03:43.369 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/53991,v1:127.0.0.1:6807/53991]", 2021-05-12T12:03:43.370 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:03:43.370 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:03:43.370 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:03:43.370 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:03:43.371 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:03:43.371 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:03:43.371 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:03:43.371 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:03:43.371 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:03:43.372 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:03:43.372 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:03:43.372 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:03:43.372 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:03:43.372 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.373 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.373 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:03:43.373 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:03:43.373 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/48511,v1:127.0.0.1:6813/48511]", 2021-05-12T12:03:43.373 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:03:43.374 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:03:43.374 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:03:43.374 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:03:43.374 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:03:43.375 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:03:43.375 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:03:43.375 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:03:43.375 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:03:43.375 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:03:43.376 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:03:43.376 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:03:43.376 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:03:43.376 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:03:43.376 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:03:43.377 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:03:43.377 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:03:43.377 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:03:43.377 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:03:43.378 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:03:43.378 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:03:43.378 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:03:43.378 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:03:43.378 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:03:43.379 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:03:43.379 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/48511,v1:127.0.0.1:6811/48511]", 2021-05-12T12:03:43.379 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:03:43.379 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48511,v1:127.0.0.1:6817/48511]", 2021-05-12T12:03:43.379 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48511,v1:127.0.0.1:6815/48511]", 2021-05-12T12:03:43.380 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:03:43.380 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:03:43.380 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:03:43.380 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:03:43.381 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:03:43.381 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:03:43.381 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:03:43.381 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:03:43.381 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:03:43.382 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:03:43.382 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:03:43.382 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:03:43.382 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:03:43.382 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.383 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.383 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:03:43.383 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:03:43.383 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/54548,v1:127.0.0.1:6813/54548]", 2021-05-12T12:03:43.384 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:03:43.384 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:03:43.384 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:03:43.384 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:03:43.384 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:03:43.385 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:03:43.385 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:03:43.385 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:03:43.385 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:03:43.385 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:03:43.386 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:03:43.386 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:03:43.386 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:03:43.386 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:03:43.386 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:03:43.387 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:03:43.387 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:03:43.387 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:03:43.387 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:03:43.388 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:03:43.388 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:03:43.388 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:03:43.388 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:03:43.388 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:03:43.389 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:03:43.389 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/54548,v1:127.0.0.1:6811/54548]", 2021-05-12T12:03:43.389 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:03:43.389 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/54548,v1:127.0.0.1:6817/54548]", 2021-05-12T12:03:43.389 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/54548,v1:127.0.0.1:6815/54548]", 2021-05-12T12:03:43.390 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:03:43.390 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:03:43.390 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:03:43.390 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:03:43.391 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:03:43.391 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:03:43.391 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:03:43.391 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:03:43.391 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:03:43.392 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:03:43.392 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:03:43.392 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:03:43.392 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:03:43.392 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.393 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.393 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_clean_epochs": { 2021-05-12T12:03:43.393 INFO:tasks.workunit.client.0.smithi191.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:03:43.393 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": { 2021-05-12T12:03:43.393 INFO:tasks.workunit.client.0.smithi191.stdout: "per_pool": [] 2021-05-12T12:03:43.394 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.394 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_epochs": [ 2021-05-12T12:03:43.394 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.394 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:03:43.394 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 34 2021-05-12T12:03:43.395 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.395 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.395 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:03:43.395 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 10 2021-05-12T12:03:43.395 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.396 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.396 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:03:43.396 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 36 2021-05-12T12:03:43.396 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.396 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.397 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.397 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_first_committed": 1, 2021-05-12T12:03:43.397 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_last_committed": 39, 2021-05-12T12:03:43.397 INFO:tasks.workunit.client.0.smithi191.stdout: "crushmap": { 2021-05-12T12:03:43.397 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": [ 2021-05-12T12:03:43.398 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.398 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:03:43.398 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.0", 2021-05-12T12:03:43.398 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:03:43.399 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.399 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.399 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:03:43.399 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.1", 2021-05-12T12:03:43.399 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:03:43.400 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.400 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.400 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:03:43.400 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.2", 2021-05-12T12:03:43.400 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:03:43.401 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.401 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.401 INFO:tasks.workunit.client.0.smithi191.stdout: "types": [ 2021-05-12T12:03:43.401 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.402 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 0, 2021-05-12T12:03:43.402 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd" 2021-05-12T12:03:43.402 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.402 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.402 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:03:43.403 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "host" 2021-05-12T12:03:43.403 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.403 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.403 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 2, 2021-05-12T12:03:43.403 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "chassis" 2021-05-12T12:03:43.404 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.404 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.404 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 3, 2021-05-12T12:03:43.404 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "rack" 2021-05-12T12:03:43.405 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.405 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.405 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 4, 2021-05-12T12:03:43.405 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "row" 2021-05-12T12:03:43.405 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.406 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.406 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 5, 2021-05-12T12:03:43.406 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pdu" 2021-05-12T12:03:43.406 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.406 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.407 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 6, 2021-05-12T12:03:43.407 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pod" 2021-05-12T12:03:43.407 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.407 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.408 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 7, 2021-05-12T12:03:43.408 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "room" 2021-05-12T12:03:43.408 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.408 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.408 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 8, 2021-05-12T12:03:43.409 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "datacenter" 2021-05-12T12:03:43.409 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.409 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.409 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 9, 2021-05-12T12:03:43.409 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "zone" 2021-05-12T12:03:43.410 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.410 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.410 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 10, 2021-05-12T12:03:43.410 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "region" 2021-05-12T12:03:43.410 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.411 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.411 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:03:43.411 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "root" 2021-05-12T12:03:43.411 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.411 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.412 INFO:tasks.workunit.client.0.smithi191.stdout: "buckets": [ 2021-05-12T12:03:43.412 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.412 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -1, 2021-05-12T12:03:43.412 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default", 2021-05-12T12:03:43.412 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:03:43.413 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:03:43.413 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:03:43.413 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:03:43.413 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:03:43.413 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:03:43.414 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.414 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:03:43.414 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:03:43.414 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:03:43.414 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.415 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.415 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.415 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.415 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -2, 2021-05-12T12:03:43.416 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default~hdd", 2021-05-12T12:03:43.416 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:03:43.416 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:03:43.416 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:03:43.416 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:03:43.417 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:03:43.417 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:03:43.417 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.417 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:03:43.417 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:03:43.418 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:03:43.418 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.418 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.418 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.418 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.419 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:03:43.419 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191", 2021-05-12T12:03:43.419 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:03:43.419 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:03:43.419 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:03:43.420 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:03:43.420 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:03:43.420 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:03:43.420 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.420 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:03:43.421 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:03:43.421 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:03:43.421 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.421 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.421 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:03:43.422 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:03:43.422 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:03:43.422 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.422 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.422 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:03:43.423 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:03:43.423 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:03:43.423 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.423 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.423 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.424 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.424 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:03:43.424 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191~hdd", 2021-05-12T12:03:43.424 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:03:43.424 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:03:43.425 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:03:43.425 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:03:43.425 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:03:43.425 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:03:43.426 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.426 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:03:43.426 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:03:43.426 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:03:43.426 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.427 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.427 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:03:43.427 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:03:43.427 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:03:43.427 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.428 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.428 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:03:43.428 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:03:43.428 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:03:43.428 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.429 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.429 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.429 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.429 INFO:tasks.workunit.client.0.smithi191.stdout: "rules": [ 2021-05-12T12:03:43.429 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.430 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 0, 2021-05-12T12:03:43.430 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "replicated_rule", 2021-05-12T12:03:43.430 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 0, 2021-05-12T12:03:43.430 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:03:43.430 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 1, 2021-05-12T12:03:43.431 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 10, 2021-05-12T12:03:43.431 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2021-05-12T12:03:43.431 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.431 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2021-05-12T12:03:43.431 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2021-05-12T12:03:43.432 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2021-05-12T12:03:43.432 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.432 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.432 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_firstn", 2021-05-12T12:03:43.432 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2021-05-12T12:03:43.433 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2021-05-12T12:03:43.433 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.433 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.433 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2021-05-12T12:03:43.433 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.434 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:03:43.434 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.434 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.434 INFO:tasks.workunit.client.0.smithi191.stdout: "tunables": { 2021-05-12T12:03:43.435 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_tries": 0, 2021-05-12T12:03:43.435 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:03:43.435 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_total_tries": 50, 2021-05-12T12:03:43.435 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:03:43.435 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:03:43.436 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_stable": 1, 2021-05-12T12:03:43.436 INFO:tasks.workunit.client.0.smithi191.stdout: "straw_calc_version": 1, 2021-05-12T12:03:43.436 INFO:tasks.workunit.client.0.smithi191.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:03:43.436 INFO:tasks.workunit.client.0.smithi191.stdout: "profile": "jewel", 2021-05-12T12:03:43.437 INFO:tasks.workunit.client.0.smithi191.stdout: "optimal_tunables": 1, 2021-05-12T12:03:43.437 INFO:tasks.workunit.client.0.smithi191.stdout: "legacy_tunables": 0, 2021-05-12T12:03:43.437 INFO:tasks.workunit.client.0.smithi191.stdout: "minimum_required_version": "jewel", 2021-05-12T12:03:43.437 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables": 1, 2021-05-12T12:03:43.437 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables2": 1, 2021-05-12T12:03:43.438 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v2_rules": 0, 2021-05-12T12:03:43.438 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables3": 1, 2021-05-12T12:03:43.438 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v3_rules": 0, 2021-05-12T12:03:43.438 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v4_buckets": 1, 2021-05-12T12:03:43.438 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables5": 1, 2021-05-12T12:03:43.439 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v5_rules": 0 2021-05-12T12:03:43.439 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.439 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_args": {} 2021-05-12T12:03:43.439 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.439 INFO:tasks.workunit.client.0.smithi191.stdout: "fsmap": { 2021-05-12T12:03:43.440 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:03:43.440 INFO:tasks.workunit.client.0.smithi191.stdout: "default_fscid": -1, 2021-05-12T12:03:43.440 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": { 2021-05-12T12:03:43.440 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": {}, 2021-05-12T12:03:43.440 INFO:tasks.workunit.client.0.smithi191.stdout: "ro_compat": {}, 2021-05-12T12:03:43.441 INFO:tasks.workunit.client.0.smithi191.stdout: "incompat": { 2021-05-12T12:03:43.441 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_1": "base v0.20", 2021-05-12T12:03:43.441 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:03:43.441 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:03:43.441 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:03:43.442 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:03:43.442 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:03:43.442 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_8": "no anchor table", 2021-05-12T12:03:43.442 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_9": "file layout v2", 2021-05-12T12:03:43.443 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:03:43.443 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.443 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.443 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_flags": { 2021-05-12T12:03:43.443 INFO:tasks.workunit.client.0.smithi191.stdout: "enable_multiple": true, 2021-05-12T12:03:43.444 INFO:tasks.workunit.client.0.smithi191.stdout: "ever_enabled_multiple": true 2021-05-12T12:03:43.444 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.444 INFO:tasks.workunit.client.0.smithi191.stdout: "standbys": [], 2021-05-12T12:03:43.444 INFO:tasks.workunit.client.0.smithi191.stdout: "filesystems": [] 2021-05-12T12:03:43.444 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.445 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:03:43.445 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:03:43.445 INFO:tasks.workunit.client.0.smithi191.stdout: "auth": { 2021-05-12T12:03:43.445 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:03:43.445 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 5, 2021-05-12T12:03:43.446 INFO:tasks.workunit.client.0.smithi191.stdout: "num_secrets": 3 2021-05-12T12:03:43.446 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.446 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:03:43.446 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_active": 1, 2021-05-12T12:03:43.446 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_unknown": 0, 2021-05-12T12:03:43.447 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osd": 3, 2021-05-12T12:03:43.447 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_sum": { 2021-05-12T12:03:43.447 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:03:43.447 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48300, 2021-05-12T12:03:43.448 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:03:43.448 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:03:43.448 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:03:43.448 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:03:43.448 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:03:43.449 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:03:43.449 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:03:43.449 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:03:43.449 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:03:43.449 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:03:43.450 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:03:43.450 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:03:43.450 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 100, 2021-05-12T12:03:43.450 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 100, 2021-05-12T12:03:43.450 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:03:43.451 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:03:43.451 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:03:43.451 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:03:43.451 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:03:43.451 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:03:43.452 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:03:43.452 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:03:43.452 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:03:43.452 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:03:43.452 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:03:43.453 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:03:43.453 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:03:43.453 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:03:43.453 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:03:43.454 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:03:43.454 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:03:43.454 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:03:43.454 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:03:43.454 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:03:43.455 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:03:43.455 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:03:43.455 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:03:43.455 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:03:43.455 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:03:43.456 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.456 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:03:43.456 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:03:43.456 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:03:43.457 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:03:43.457 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 0, 2021-05-12T12:03:43.457 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 0, 2021-05-12T12:03:43.457 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:03:43.457 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:03:43.458 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:03:43.458 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:03:43.458 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:03:43.458 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.458 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 16, 2021-05-12T12:03:43.459 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 16, 2021-05-12T12:03:43.459 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 2, 2021-05-12T12:03:43.459 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 2, 2021-05-12T12:03:43.459 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 0 2021-05-12T12:03:43.459 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.460 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum": { 2021-05-12T12:03:43.460 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:03:43.460 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:03:43.460 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:03:43.461 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:03:43.461 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:03:43.461 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:03:43.461 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:03:43.461 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 18408, 2021-05-12T12:03:43.462 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2056, 2021-05-12T12:03:43.462 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:03:43.462 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16317, 2021-05-12T12:03:43.462 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314554392, 2021-05-12T12:03:43.462 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:03:43.463 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:03:43.463 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322103697408, 2021-05-12T12:03:43.463 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:03:43.463 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2105344, 2021-05-12T12:03:43.464 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 404185, 2021-05-12T12:03:43.464 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:03:43.464 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:03:43.464 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:03:43.464 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:03:43.465 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 16709520 2021-05-12T12:03:43.465 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.465 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:03:43.465 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:03:43.465 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:03:43.466 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:03:43.466 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:03:43.466 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:03:43.466 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:03:43.466 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.467 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:03:43.467 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 0, 2021-05-12T12:03:43.467 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 0, 2021-05-12T12:03:43.467 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 0, 2021-05-12T12:03:43.467 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 0 2021-05-12T12:03:43.468 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.468 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:03:43.468 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:03:43.468 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.468 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum_by_class": { 2021-05-12T12:03:43.469 INFO:tasks.workunit.client.0.smithi191.stdout: "hdd": { 2021-05-12T12:03:43.469 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:03:43.469 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:03:43.469 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:03:43.469 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:03:43.470 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:03:43.470 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:03:43.470 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:03:43.470 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 18408, 2021-05-12T12:03:43.471 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2056, 2021-05-12T12:03:43.471 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:03:43.471 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16317, 2021-05-12T12:03:43.471 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314554392, 2021-05-12T12:03:43.471 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:03:43.472 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:03:43.472 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322103697408, 2021-05-12T12:03:43.472 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:03:43.472 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2105344, 2021-05-12T12:03:43.472 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 404185, 2021-05-12T12:03:43.473 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:03:43.473 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:03:43.473 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:03:43.473 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:03:43.474 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 16709520 2021-05-12T12:03:43.474 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.474 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:03:43.474 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:03:43.474 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:03:43.475 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:03:43.475 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:03:43.475 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:03:43.475 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:03:43.475 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.476 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:03:43.476 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 0, 2021-05-12T12:03:43.476 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 0, 2021-05-12T12:03:43.476 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 0, 2021-05-12T12:03:43.476 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 0 2021-05-12T12:03:43.477 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.477 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:03:43.477 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:03:43.477 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.477 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.478 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_stats": [ 2021-05-12T12:03:43.478 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.478 INFO:tasks.workunit.client.0.smithi191.stdout: "poolid": 1, 2021-05-12T12:03:43.478 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:03:43.479 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:03:43.479 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48300, 2021-05-12T12:03:43.479 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:03:43.479 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:03:43.479 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:03:43.480 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:03:43.480 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:03:43.480 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:03:43.480 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:03:43.480 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:03:43.481 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:03:43.481 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:03:43.481 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:03:43.481 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:03:43.481 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 100, 2021-05-12T12:03:43.482 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 100, 2021-05-12T12:03:43.482 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:03:43.482 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:03:43.482 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:03:43.482 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:03:43.483 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:03:43.483 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:03:43.483 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:03:43.483 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:03:43.483 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:03:43.484 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:03:43.484 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:03:43.484 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:03:43.484 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:03:43.484 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:03:43.485 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:03:43.485 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:03:43.485 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:03:43.485 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:03:43.486 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:03:43.486 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:03:43.486 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:03:43.486 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:03:43.486 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:03:43.487 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:03:43.487 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:03:43.487 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.487 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:03:43.487 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:03:43.488 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:03:43.488 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:03:43.488 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 1220608, 2021-05-12T12:03:43.488 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 143228, 2021-05-12T12:03:43.488 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:03:43.489 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:03:43.489 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:03:43.489 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:03:43.489 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:03:43.489 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.490 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 16, 2021-05-12T12:03:43.490 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 16, 2021-05-12T12:03:43.490 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 2, 2021-05-12T12:03:43.490 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 2, 2021-05-12T12:03:43.491 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 3 2021-05-12T12:03:43.491 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.491 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.491 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_stats": [ 2021-05-12T12:03:43.491 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.492 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:03:43.492 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 146028888075 2021-05-12T12:03:43.492 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.492 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.492 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:03:43.493 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 42949672977 2021-05-12T12:03:43.493 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.493 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.493 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:03:43.493 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 154618822666 2021-05-12T12:03:43.494 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.494 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.494 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_state": [ 2021-05-12T12:03:43.494 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.494 INFO:tasks.workunit.client.0.smithi191.stdout: "state": "active+undersized+degraded", 2021-05-12T12:03:43.495 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 1 2021-05-12T12:03:43.495 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.495 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.495 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_osd": [ 2021-05-12T12:03:43.496 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.496 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:03:43.496 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:03:43.496 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:03:43.496 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:03:43.497 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.497 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.497 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:03:43.497 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:03:43.497 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 0, 2021-05-12T12:03:43.498 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:03:43.498 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.498 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.498 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:03:43.499 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 1, 2021-05-12T12:03:43.499 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:03:43.499 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:03:43.499 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.499 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.500 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": [ 2021-05-12T12:03:43.500 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:03:43.500 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:03:43.500 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": {} 2021-05-12T12:03:43.501 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.501 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:03:43.501 INFO:tasks.workunit.client.0.smithi191.stdout: "servicemap": { 2021-05-12T12:03:43.501 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 4, 2021-05-12T12:03:43.501 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:03:03.437094+0000", 2021-05-12T12:03:43.502 INFO:tasks.workunit.client.0.smithi191.stdout: "services": { 2021-05-12T12:03:43.502 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": { 2021-05-12T12:03:43.502 INFO:tasks.workunit.client.0.smithi191.stdout: "daemons": { 2021-05-12T12:03:43.502 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": "", 2021-05-12T12:03:43.502 INFO:tasks.workunit.client.0.smithi191.stdout: "0": { 2021-05-12T12:03:43.503 INFO:tasks.workunit.client.0.smithi191.stdout: "start_epoch": 0, 2021-05-12T12:03:43.503 INFO:tasks.workunit.client.0.smithi191.stdout: "start_stamp": "0.000000", 2021-05-12T12:03:43.503 INFO:tasks.workunit.client.0.smithi191.stdout: "gid": 0, 2021-05-12T12:03:43.503 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:03:43.503 INFO:tasks.workunit.client.0.smithi191.stdout: "metadata": {}, 2021-05-12T12:03:43.504 INFO:tasks.workunit.client.0.smithi191.stdout: "task_status": {} 2021-05-12T12:03:43.504 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.504 INFO:tasks.workunit.client.0.smithi191.stdout: "2": { 2021-05-12T12:03:43.504 INFO:tasks.workunit.client.0.smithi191.stdout: "start_epoch": 0, 2021-05-12T12:03:43.504 INFO:tasks.workunit.client.0.smithi191.stdout: "start_stamp": "0.000000", 2021-05-12T12:03:43.505 INFO:tasks.workunit.client.0.smithi191.stdout: "gid": 0, 2021-05-12T12:03:43.505 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:03:43.505 INFO:tasks.workunit.client.0.smithi191.stdout: "metadata": {}, 2021-05-12T12:03:43.505 INFO:tasks.workunit.client.0.smithi191.stdout: "task_status": {} 2021-05-12T12:03:43.505 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.506 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.506 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.506 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.506 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:03:43.506 INFO:tasks.workunit.client.0.smithi191.stdout: "paxos": { 2021-05-12T12:03:43.507 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:03:43.507 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 367, 2021-05-12T12:03:43.507 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pn": 0, 2021-05-12T12:03:43.507 INFO:tasks.workunit.client.0.smithi191.stdout: "accepted_pn": 0 2021-05-12T12:03:43.507 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:03:43.508 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:03:43.508 INFO:tasks.workunit.client.0.smithi191.stderr:report 3874782445 2021-05-12T12:03:43.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:03:43.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2021-05-12T12:03:43.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2021-05-12T12:03:43.510 INFO:tasks.workunit.client.0.smithi191.stdout:writing non-divergent object existing_4 2021-05-12T12:03:43.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2021-05-12T12:03:43.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:03:43.612 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:03:43.612 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:03:07.997934+0000 27'100 39:122 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:03:43.612 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:03:43.613 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:03:43.614 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:03:43.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T12:03:43.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:03:43.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2021-05-12T12:03:43.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:03:43.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:03:43.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:03:43.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:03:43.731 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:03:43.731 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:03:43.732 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:03:43.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:03:43.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:03:43.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:03:43.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:03:43.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:03:43.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:03:43.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:03:43.786 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:03:44.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:03:44.152 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:03:44.152 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:03:44.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:03:44.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:44.153 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:03:44.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=146028888077 2021-05-12T12:03:44.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 146028888077 2021-05-12T12:03:44.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-146028888077' 2021-05-12T12:03:44.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:44.300 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:03:44.396 INFO:tasks.workunit.client.0.smithi191.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:03:44.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:03:44.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:03:44.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:03:44.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:44.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:03:44.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=154618822668 2021-05-12T12:03:44.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 154618822668 2021-05-12T12:03:44.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-146028888077 2-154618822668' 2021-05-12T12:03:44.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:03:44.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-146028888077 2021-05-12T12:03:44.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:03:44.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:03:44.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-146028888077 2021-05-12T12:03:44.535 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:03:44.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=146028888077 2021-05-12T12:03:44.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 146028888077' 2021-05-12T12:03:44.535 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 146028888077 2021-05-12T12:03:44.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:03:44.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 146028888075 -lt 146028888077 2021-05-12T12:03:44.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:03:45.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:03:45.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:03:46.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 146028888077 -lt 146028888077 2021-05-12T12:03:46.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:03:46.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822668 2021-05-12T12:03:46.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:03:46.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:03:46.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-154618822668 2021-05-12T12:03:46.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:03:46.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=154618822668 2021-05-12T12:03:46.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 154618822668' 2021-05-12T12:03:46.277 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 154618822668 2021-05-12T12:03:46.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:03:46.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 154618822669 -lt 154618822668 2021-05-12T12:03:46.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:03:46.644 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:46.644 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:47.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:03:47.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:47.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:47.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:47.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:47.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:47.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:47.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:47.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:47.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:47.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:47.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:48.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:48.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:03:48.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:03:48.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:03:48.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:48.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:03:48.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:48.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:48.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:48.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:48.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:48.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:48.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:48.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:48.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:48.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:48.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:48.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:48.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:48.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:48.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:48.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:49.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:49.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:49.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:03:49.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:49.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:03:49.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:49.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:49.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:49.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:49.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:49.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:49.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:49.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:49.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:49.976 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:49.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:49.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:50.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:50.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:50.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:50.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:50.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:50.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:50.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:50.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:50.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:50.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:50.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:50.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:03:50.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:50.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:03:51.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:51.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:51.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:51.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:51.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:51.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:51.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:51.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:51.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:51.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:51.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:51.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:52.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:52.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:52.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:52.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:52.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:52.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:52.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:52.184 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:52.184 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:52.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:52.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:52.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:03:52.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:52.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:03:53.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:53.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:53.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:53.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:53.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:53.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:53.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:53.464 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:53.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:53.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:53.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:53.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:54.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:54.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:54.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:54.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:54.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:54.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:54.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:54.305 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:54.305 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:54.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:54.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:54.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:03:54.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:54.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:03:56.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:56.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:56.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:56.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:56.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:56.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:56.390 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:56.390 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:56.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:56.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:56.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:56.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:57.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:57.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:57.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:57.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:57.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:57.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:57.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:57.237 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:57.237 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:57.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:57.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:57.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:03:57.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:57.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:04:00.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:00.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:00.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:00.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:00.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:00.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:00.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:00.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:01.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:01.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:01.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:01.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:01.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:01.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:01.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:01.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:01.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:01.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:01.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:01.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:01.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:02.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:02.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:02.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:04:02.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:02.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:04:08.654 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:08.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:08.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:08.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:08.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:08.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:08.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:08.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:09.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:09.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:09.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:09.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:09.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:09.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:09.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:09.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:09.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:09.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:09.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:09.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:09.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:09.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:09.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:09.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:04:09.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:09.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:04:17.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:17.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:17.282 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:17.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:17.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:17.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:17.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:17.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:17.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:17.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:17.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:17.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:18.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:18.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:18.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:18.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:18.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:18.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:18.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:18.176 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:18.176 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:18.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:18.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:18.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:04:18.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:04:19.120 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:04:19.120 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_fingerprint": "bc472bb2-a4e4-4155-b217-c0b688d45a1a", 2021-05-12T12:04:19.121 INFO:tasks.workunit.client.0.smithi191.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:19.121 INFO:tasks.workunit.client.0.smithi191.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:04:19.121 INFO:tasks.workunit.client.0.smithi191.stdout: "timestamp": "2021-05-12T12:04:19.117218+0000", 2021-05-12T12:04:19.121 INFO:tasks.workunit.client.0.smithi191.stdout: "tag": "", 2021-05-12T12:04:19.122 INFO:tasks.workunit.client.0.smithi191.stdout: "health": { 2021-05-12T12:04:19.122 INFO:tasks.workunit.client.0.smithi191.stdout: "status": "HEALTH_ERR", 2021-05-12T12:04:19.122 INFO:tasks.workunit.client.0.smithi191.stdout: "checks": { 2021-05-12T12:04:19.122 INFO:tasks.workunit.client.0.smithi191.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:04:19.123 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:04:19.123 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:04:19.123 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:04:19.123 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:04:19.124 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.124 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:04:19.124 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.124 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:04:19.124 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.125 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.125 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:04:19.126 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.126 INFO:tasks.workunit.client.0.smithi191.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:04:19.126 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:19.126 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:04:19.127 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:04:19.127 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 17 2021-05-12T12:04:19.127 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.127 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [], 2021-05-12T12:04:19.127 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:04:19.128 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.128 INFO:tasks.workunit.client.0.smithi191.stdout: "OSD_DOWN": { 2021-05-12T12:04:19.128 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:19.128 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:04:19.128 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 osds down", 2021-05-12T12:04:19.129 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:04:19.129 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.129 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:04:19.129 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.129 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "osd.1 (root=default,host=smithi191) is down" 2021-05-12T12:04:19.130 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.130 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.130 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:04:19.130 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.131 INFO:tasks.workunit.client.0.smithi191.stdout: "PG_DEGRADED": { 2021-05-12T12:04:19.131 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:19.131 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:04:19.131 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2021-05-12T12:04:19.131 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 2 2021-05-12T12:04:19.132 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.132 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:04:19.132 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.132 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:04:19.132 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.133 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.133 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:04:19.133 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.133 INFO:tasks.workunit.client.0.smithi191.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:04:19.134 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:19.134 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:04:19.134 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:04:19.134 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:04:19.134 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.135 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:04:19.135 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.135 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:04:19.135 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.136 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.136 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:04:19.136 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.136 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.136 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:04:19.137 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.137 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.137 INFO:tasks.workunit.client.0.smithi191.stdout: "mutes": [] 2021-05-12T12:04:19.137 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.137 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_first_committed": 1, 2021-05-12T12:04:19.138 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_last_committed": 1, 2021-05-12T12:04:19.138 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap": { 2021-05-12T12:04:19.138 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:04:19.138 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:04:19.139 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:01:41.535760Z", 2021-05-12T12:04:19.139 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:01:41.535760Z", 2021-05-12T12:04:19.139 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release": 17, 2021-05-12T12:04:19.139 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:04:19.139 INFO:tasks.workunit.client.0.smithi191.stdout: "election_strategy": 1, 2021-05-12T12:04:19.140 INFO:tasks.workunit.client.0.smithi191.stdout: "disallowed_leaders: ": "", 2021-05-12T12:04:19.140 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": false, 2021-05-12T12:04:19.140 INFO:tasks.workunit.client.0.smithi191.stdout: "features": { 2021-05-12T12:04:19.140 INFO:tasks.workunit.client.0.smithi191.stdout: "persistent": [ 2021-05-12T12:04:19.140 INFO:tasks.workunit.client.0.smithi191.stdout: "kraken", 2021-05-12T12:04:19.141 INFO:tasks.workunit.client.0.smithi191.stdout: "luminous", 2021-05-12T12:04:19.141 INFO:tasks.workunit.client.0.smithi191.stdout: "mimic", 2021-05-12T12:04:19.141 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap-prune", 2021-05-12T12:04:19.141 INFO:tasks.workunit.client.0.smithi191.stdout: "nautilus", 2021-05-12T12:04:19.141 INFO:tasks.workunit.client.0.smithi191.stdout: "octopus", 2021-05-12T12:04:19.142 INFO:tasks.workunit.client.0.smithi191.stdout: "pacific", 2021-05-12T12:04:19.142 INFO:tasks.workunit.client.0.smithi191.stdout: "elector-pinging", 2021-05-12T12:04:19.142 INFO:tasks.workunit.client.0.smithi191.stdout: "quincy" 2021-05-12T12:04:19.142 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.143 INFO:tasks.workunit.client.0.smithi191.stdout: "optional": [] 2021-05-12T12:04:19.143 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.143 INFO:tasks.workunit.client.0.smithi191.stdout: "mons": [ 2021-05-12T12:04:19.143 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.143 INFO:tasks.workunit.client.0.smithi191.stdout: "rank": 0, 2021-05-12T12:04:19.144 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "a", 2021-05-12T12:04:19.144 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:04:19.144 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.144 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.144 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.145 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:04:19.145 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 0 2021-05-12T12:04:19.145 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.145 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.145 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.146 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:04:19.146 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:04:19.146 INFO:tasks.workunit.client.0.smithi191.stdout: "priority": 0, 2021-05-12T12:04:19.146 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 0, 2021-05-12T12:04:19.146 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_location": "{}" 2021-05-12T12:04:19.147 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.147 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.147 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.147 INFO:tasks.workunit.client.0.smithi191.stdout: "quorum": [ 2021-05-12T12:04:19.148 INFO:tasks.workunit.client.0.smithi191.stdout: 0 2021-05-12T12:04:19.148 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.148 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap": { 2021-05-12T12:04:19.148 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 39, 2021-05-12T12:04:19.148 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:04:19.149 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:01:42.410837+0000", 2021-05-12T12:04:19.149 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:03:07.662094+0000", 2021-05-12T12:04:19.149 INFO:tasks.workunit.client.0.smithi191.stdout: "last_up_change": "2021-05-12T12:03:05.079814+0000", 2021-05-12T12:04:19.149 INFO:tasks.workunit.client.0.smithi191.stdout: "last_in_change": "2021-05-12T12:03:07.343642+0000", 2021-05-12T12:04:19.149 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:04:19.150 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_num": 5800384, 2021-05-12T12:04:19.150 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_set": [ 2021-05-12T12:04:19.150 INFO:tasks.workunit.client.0.smithi191.stdout: "nodown", 2021-05-12T12:04:19.150 INFO:tasks.workunit.client.0.smithi191.stdout: "noin", 2021-05-12T12:04:19.150 INFO:tasks.workunit.client.0.smithi191.stdout: "noout", 2021-05-12T12:04:19.151 INFO:tasks.workunit.client.0.smithi191.stdout: "pglog_hardlimit", 2021-05-12T12:04:19.151 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snapdirs", 2021-05-12T12:04:19.151 INFO:tasks.workunit.client.0.smithi191.stdout: "recovery_deletes", 2021-05-12T12:04:19.151 INFO:tasks.workunit.client.0.smithi191.stdout: "sortbitwise" 2021-05-12T12:04:19.152 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.152 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_version": 9, 2021-05-12T12:04:19.152 INFO:tasks.workunit.client.0.smithi191.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:04:19.152 INFO:tasks.workunit.client.0.smithi191.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:04:19.152 INFO:tasks.workunit.client.0.smithi191.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:04:19.153 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_snapshot": "", 2021-05-12T12:04:19.153 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_max": 1, 2021-05-12T12:04:19.153 INFO:tasks.workunit.client.0.smithi191.stdout: "max_osd": 3, 2021-05-12T12:04:19.153 INFO:tasks.workunit.client.0.smithi191.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:04:19.153 INFO:tasks.workunit.client.0.smithi191.stdout: "min_compat_client": "jewel", 2021-05-12T12:04:19.154 INFO:tasks.workunit.client.0.smithi191.stdout: "require_osd_release": "quincy", 2021-05-12T12:04:19.154 INFO:tasks.workunit.client.0.smithi191.stdout: "pools": [ 2021-05-12T12:04:19.154 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.154 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:04:19.155 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_name": "test", 2021-05-12T12:04:19.155 INFO:tasks.workunit.client.0.smithi191.stdout: "create_time": "2021-05-12T12:02:13.813588+0000", 2021-05-12T12:04:19.155 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": 1, 2021-05-12T12:04:19.155 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_names": "hashpspool", 2021-05-12T12:04:19.155 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:04:19.156 INFO:tasks.workunit.client.0.smithi191.stdout: "size": 3, 2021-05-12T12:04:19.156 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 2, 2021-05-12T12:04:19.156 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_rule": 0, 2021-05-12T12:04:19.156 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:04:19.157 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:04:19.157 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:04:19.157 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:04:19.157 INFO:tasks.workunit.client.0.smithi191.stdout: "object_hash": 2, 2021-05-12T12:04:19.157 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:04:19.158 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num": 1, 2021-05-12T12:04:19.158 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num": 1, 2021-05-12T12:04:19.158 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num_target": 1, 2021-05-12T12:04:19.158 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_target": 1, 2021-05-12T12:04:19.159 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_pending": 1, 2021-05-12T12:04:19.159 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pg_merge_meta": { 2021-05-12T12:04:19.159 INFO:tasks.workunit.client.0.smithi191.stdout: "source_pgid": "0.0", 2021-05-12T12:04:19.159 INFO:tasks.workunit.client.0.smithi191.stdout: "ready_epoch": 0, 2021-05-12T12:04:19.159 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_started": 0, 2021-05-12T12:04:19.160 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": 0, 2021-05-12T12:04:19.160 INFO:tasks.workunit.client.0.smithi191.stdout: "source_version": "0'0", 2021-05-12T12:04:19.160 INFO:tasks.workunit.client.0.smithi191.stdout: "target_version": "0'0" 2021-05-12T12:04:19.160 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.161 INFO:tasks.workunit.client.0.smithi191.stdout: "last_change": "27", 2021-05-12T12:04:19.161 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend": "0", 2021-05-12T12:04:19.161 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:04:19.161 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:04:19.161 INFO:tasks.workunit.client.0.smithi191.stdout: "auid": 0, 2021-05-12T12:04:19.162 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:04:19.162 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_seq": 0, 2021-05-12T12:04:19.162 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_epoch": 0, 2021-05-12T12:04:19.162 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_snaps": [], 2021-05-12T12:04:19.162 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps": "[]", 2021-05-12T12:04:19.163 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_bytes": 0, 2021-05-12T12:04:19.163 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_objects": 0, 2021-05-12T12:04:19.163 INFO:tasks.workunit.client.0.smithi191.stdout: "tiers": [], 2021-05-12T12:04:19.163 INFO:tasks.workunit.client.0.smithi191.stdout: "tier_of": -1, 2021-05-12T12:04:19.164 INFO:tasks.workunit.client.0.smithi191.stdout: "read_tier": -1, 2021-05-12T12:04:19.164 INFO:tasks.workunit.client.0.smithi191.stdout: "write_tier": -1, 2021-05-12T12:04:19.164 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_mode": "none", 2021-05-12T12:04:19.164 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_bytes": 0, 2021-05-12T12:04:19.164 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_objects": 0, 2021-05-12T12:04:19.165 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:04:19.165 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:04:19.165 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:04:19.165 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_flush_age": 0, 2021-05-12T12:04:19.165 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_evict_age": 0, 2021-05-12T12:04:19.166 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profile": "", 2021-05-12T12:04:19.166 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_params": { 2021-05-12T12:04:19.166 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "none" 2021-05-12T12:04:19.166 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.167 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_period": 0, 2021-05-12T12:04:19.167 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_count": 0, 2021-05-12T12:04:19.167 INFO:tasks.workunit.client.0.smithi191.stdout: "use_gmt_hitset": true, 2021-05-12T12:04:19.167 INFO:tasks.workunit.client.0.smithi191.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:04:19.167 INFO:tasks.workunit.client.0.smithi191.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:04:19.168 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:04:19.168 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:04:19.168 INFO:tasks.workunit.client.0.smithi191.stdout: "grade_table": [], 2021-05-12T12:04:19.168 INFO:tasks.workunit.client.0.smithi191.stdout: "stripe_width": 0, 2021-05-12T12:04:19.168 INFO:tasks.workunit.client.0.smithi191.stdout: "expected_num_objects": 0, 2021-05-12T12:04:19.169 INFO:tasks.workunit.client.0.smithi191.stdout: "fast_read": false, 2021-05-12T12:04:19.169 INFO:tasks.workunit.client.0.smithi191.stdout: "options": {}, 2021-05-12T12:04:19.169 INFO:tasks.workunit.client.0.smithi191.stdout: "application_metadata": {} 2021-05-12T12:04:19.169 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.169 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.170 INFO:tasks.workunit.client.0.smithi191.stdout: "osds": [ 2021-05-12T12:04:19.170 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.170 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:04:19.170 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "680f25cc-1994-4806-b5f0-2bb987fe27cd", 2021-05-12T12:04:19.171 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:04:19.171 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:04:19.171 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:04:19.171 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:04:19.171 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 5, 2021-05-12T12:04:19.172 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 27, 2021-05-12T12:04:19.172 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 34, 2021-05-12T12:04:19.172 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 38, 2021-05-12T12:04:19.172 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 28, 2021-05-12T12:04:19.172 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:04:19.173 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:04:19.173 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.173 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.173 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.173 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:04:19.174 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:04:19.174 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.174 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.174 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.175 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:04:19.175 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:04:19.175 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.175 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.175 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.176 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:04:19.176 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.176 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.176 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.176 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:04:19.177 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:04:19.177 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.177 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.177 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.177 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:04:19.178 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:04:19.178 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.178 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.178 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.179 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:04:19.179 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.179 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.179 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.179 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:04:19.180 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:04:19.180 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.180 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.180 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.180 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:04:19.181 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:04:19.181 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.181 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.181 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.181 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:04:19.182 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.182 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.182 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.182 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:04:19.183 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:04:19.183 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.183 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.183 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.183 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:04:19.184 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 53991 2021-05-12T12:04:19.184 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.184 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.184 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.184 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6803/53991", 2021-05-12T12:04:19.185 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6805/53991", 2021-05-12T12:04:19.185 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6809/53991", 2021-05-12T12:04:19.185 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6807/53991", 2021-05-12T12:04:19.185 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:04:19.185 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:04:19.186 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:04:19.186 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.186 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.186 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.186 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:04:19.187 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a", 2021-05-12T12:04:19.187 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 0, 2021-05-12T12:04:19.187 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:04:19.187 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:04:19.187 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:04:19.188 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 0, 2021-05-12T12:04:19.188 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 0, 2021-05-12T12:04:19.188 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 10, 2021-05-12T12:04:19.188 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 21, 2021-05-12T12:04:19.188 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 29, 2021-05-12T12:04:19.189 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:04:19.189 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:04:19.189 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.189 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.190 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.190 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:04:19.190 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:04:19.190 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.190 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.191 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.191 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:04:19.191 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:04:19.191 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.191 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.192 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.192 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:04:19.192 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.192 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.192 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.193 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:04:19.193 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:04:19.193 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.193 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.193 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.194 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:04:19.194 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:04:19.194 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.194 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.195 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.195 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:04:19.195 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.195 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.195 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.196 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:04:19.196 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:04:19.196 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.196 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.196 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.197 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:04:19.197 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:04:19.197 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.197 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.197 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.198 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:04:19.198 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.198 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.198 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.198 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:04:19.199 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:04:19.199 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.199 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.199 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.200 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:04:19.200 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 48511 2021-05-12T12:04:19.200 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.200 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.200 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.201 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/48511", 2021-05-12T12:04:19.201 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/48511", 2021-05-12T12:04:19.201 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48511", 2021-05-12T12:04:19.201 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48511", 2021-05-12T12:04:19.201 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:04:19.202 INFO:tasks.workunit.client.0.smithi191.stdout: "exists" 2021-05-12T12:04:19.202 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.202 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.202 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.202 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:04:19.203 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "9dbbea6e-ca5b-4917-9e87-b424e890b08e", 2021-05-12T12:04:19.203 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:04:19.203 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:04:19.203 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:04:19.203 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:04:19.204 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 14, 2021-05-12T12:04:19.204 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 27, 2021-05-12T12:04:19.204 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 36, 2021-05-12T12:04:19.204 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 0, 2021-05-12T12:04:19.204 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 30, 2021-05-12T12:04:19.205 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:04:19.205 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:04:19.205 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.205 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.206 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.206 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:04:19.206 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:04:19.206 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.206 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.207 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.207 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:04:19.207 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:04:19.207 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.207 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.208 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.208 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:04:19.208 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.208 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.208 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.209 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:04:19.209 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:04:19.209 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.209 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.209 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.210 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:04:19.210 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:04:19.210 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.210 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.211 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.211 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:04:19.211 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.211 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.211 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.212 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:04:19.212 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:04:19.212 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.212 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.213 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.213 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:04:19.213 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:04:19.213 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.213 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.214 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.214 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:04:19.214 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:04:19.214 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.214 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:04:19.215 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:04:19.215 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:04:19.215 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.215 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.215 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:04:19.216 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:04:19.216 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 54548 2021-05-12T12:04:19.216 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.216 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.217 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.217 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/54548", 2021-05-12T12:04:19.217 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/54548", 2021-05-12T12:04:19.217 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/54548", 2021-05-12T12:04:19.217 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/54548", 2021-05-12T12:04:19.217 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:04:19.218 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:04:19.218 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:04:19.218 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.218 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.219 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.219 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_xinfo": [ 2021-05-12T12:04:19.219 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.219 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:04:19.219 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:02:55.145006+0000", 2021-05-12T12:04:19.220 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:04:19.220 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:04:19.220 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:04:19.220 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:04:19.220 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:01:52.281020+0000", 2021-05-12T12:04:19.221 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:04:19.221 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.221 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.221 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:04:19.221 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:02:55.701124+0000", 2021-05-12T12:04:19.222 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:04:19.222 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:04:19.222 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:04:19.222 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:04:19.223 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:01.884752+0000", 2021-05-12T12:04:19.223 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:04:19.223 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.223 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.223 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:04:19.224 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:02:56.304965+0000", 2021-05-12T12:04:19.224 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:04:19.224 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:04:19.224 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:04:19.224 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:04:19.225 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:11.506727+0000", 2021-05-12T12:04:19.225 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:04:19.225 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.225 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.226 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap": [], 2021-05-12T12:04:19.226 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap_items": [], 2021-05-12T12:04:19.226 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_temp": [], 2021-05-12T12:04:19.226 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_temp": [], 2021-05-12T12:04:19.226 INFO:tasks.workunit.client.0.smithi191.stdout: "blocklist": {}, 2021-05-12T12:04:19.227 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profiles": { 2021-05-12T12:04:19.227 INFO:tasks.workunit.client.0.smithi191.stdout: "default": { 2021-05-12T12:04:19.227 INFO:tasks.workunit.client.0.smithi191.stdout: "k": "2", 2021-05-12T12:04:19.227 INFO:tasks.workunit.client.0.smithi191.stdout: "m": "2", 2021-05-12T12:04:19.227 INFO:tasks.workunit.client.0.smithi191.stdout: "plugin": "jerasure", 2021-05-12T12:04:19.227 INFO:tasks.workunit.client.0.smithi191.stdout: "technique": "reed_sol_van" 2021-05-12T12:04:19.228 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.228 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.228 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps_queue": [], 2021-05-12T12:04:19.228 INFO:tasks.workunit.client.0.smithi191.stdout: "new_removed_snaps": [], 2021-05-12T12:04:19.229 INFO:tasks.workunit.client.0.smithi191.stdout: "new_purged_snaps": [], 2021-05-12T12:04:19.229 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_node_flags": {}, 2021-05-12T12:04:19.229 INFO:tasks.workunit.client.0.smithi191.stdout: "device_class_flags": {}, 2021-05-12T12:04:19.229 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": { 2021-05-12T12:04:19.229 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_enabled": false, 2021-05-12T12:04:19.230 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_bucket_count": 0, 2021-05-12T12:04:19.230 INFO:tasks.workunit.client.0.smithi191.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:04:19.230 INFO:tasks.workunit.client.0.smithi191.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:04:19.230 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_bucket": 0 2021-05-12T12:04:19.230 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.231 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.231 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_metadata": [ 2021-05-12T12:04:19.231 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.231 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:04:19.231 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:04:19.232 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6804/53991,v1:127.0.0.1:6805/53991]", 2021-05-12T12:04:19.232 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:04:19.232 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:04:19.232 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:04:19.232 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:04:19.233 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:04:19.233 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:04:19.233 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:04:19.233 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:04:19.233 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:04:19.234 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:04:19.234 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:04:19.234 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:04:19.234 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:04:19.235 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:04:19.235 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:04:19.235 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:04:19.235 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:19.235 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:04:19.236 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:04:19.236 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:04:19.236 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:04:19.236 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:04:19.237 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:04:19.237 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:04:19.237 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:04:19.237 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6802/53991,v1:127.0.0.1:6803/53991]", 2021-05-12T12:04:19.237 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:04:19.238 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/53991,v1:127.0.0.1:6809/53991]", 2021-05-12T12:04:19.238 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/53991,v1:127.0.0.1:6807/53991]", 2021-05-12T12:04:19.238 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:04:19.238 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:04:19.238 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:04:19.239 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:04:19.239 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:04:19.239 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:04:19.239 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:04:19.239 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:04:19.240 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:04:19.240 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:04:19.240 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:04:19.240 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:04:19.240 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:04:19.241 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.241 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.241 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:04:19.241 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:04:19.242 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/48511,v1:127.0.0.1:6813/48511]", 2021-05-12T12:04:19.242 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:04:19.242 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:04:19.242 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:04:19.242 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:04:19.243 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:04:19.243 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:04:19.243 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:04:19.243 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:04:19.243 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:04:19.244 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:04:19.244 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:04:19.244 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:04:19.244 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:04:19.245 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:04:19.245 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:04:19.245 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:04:19.245 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:19.245 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:04:19.246 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:04:19.246 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:04:19.246 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:04:19.246 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:04:19.246 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:04:19.247 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:04:19.247 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:04:19.247 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/48511,v1:127.0.0.1:6811/48511]", 2021-05-12T12:04:19.247 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:04:19.248 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48511,v1:127.0.0.1:6817/48511]", 2021-05-12T12:04:19.248 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48511,v1:127.0.0.1:6815/48511]", 2021-05-12T12:04:19.248 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:04:19.248 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:04:19.248 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:04:19.249 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:04:19.249 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:04:19.249 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:04:19.249 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:04:19.249 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:04:19.250 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:04:19.250 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:04:19.250 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:04:19.250 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:04:19.250 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:04:19.251 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.251 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.251 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:04:19.251 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:04:19.251 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/54548,v1:127.0.0.1:6813/54548]", 2021-05-12T12:04:19.252 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:04:19.252 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:04:19.252 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:04:19.252 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:04:19.252 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:04:19.253 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:04:19.253 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:04:19.253 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:04:19.253 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:04:19.253 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:04:19.254 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:04:19.254 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:04:19.254 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:04:19.254 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:04:19.254 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:04:19.255 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:04:19.255 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:19.255 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:04:19.255 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:04:19.255 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:04:19.256 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:04:19.256 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:04:19.256 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:04:19.256 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:04:19.256 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:04:19.257 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/54548,v1:127.0.0.1:6811/54548]", 2021-05-12T12:04:19.257 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:04:19.257 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/54548,v1:127.0.0.1:6817/54548]", 2021-05-12T12:04:19.257 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/54548,v1:127.0.0.1:6815/54548]", 2021-05-12T12:04:19.258 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:04:19.258 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:04:19.258 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:04:19.258 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:04:19.258 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:04:19.259 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:04:19.259 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:04:19.259 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:04:19.259 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:04:19.260 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:04:19.260 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:04:19.260 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:04:19.260 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:04:19.260 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.261 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.261 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_clean_epochs": { 2021-05-12T12:04:19.261 INFO:tasks.workunit.client.0.smithi191.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:04:19.261 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": { 2021-05-12T12:04:19.262 INFO:tasks.workunit.client.0.smithi191.stdout: "per_pool": [] 2021-05-12T12:04:19.262 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.262 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_epochs": [ 2021-05-12T12:04:19.262 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.262 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:04:19.263 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 34 2021-05-12T12:04:19.263 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.263 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.263 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:04:19.263 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 10 2021-05-12T12:04:19.264 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.264 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.264 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:04:19.264 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 36 2021-05-12T12:04:19.265 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.265 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.265 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.265 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_first_committed": 1, 2021-05-12T12:04:19.265 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_last_committed": 39, 2021-05-12T12:04:19.266 INFO:tasks.workunit.client.0.smithi191.stdout: "crushmap": { 2021-05-12T12:04:19.266 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": [ 2021-05-12T12:04:19.266 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.266 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:04:19.267 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.0", 2021-05-12T12:04:19.267 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:04:19.267 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.267 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.267 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:04:19.268 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.1", 2021-05-12T12:04:19.268 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:04:19.268 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.268 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.268 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:04:19.269 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.2", 2021-05-12T12:04:19.269 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:04:19.269 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.269 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.269 INFO:tasks.workunit.client.0.smithi191.stdout: "types": [ 2021-05-12T12:04:19.270 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.270 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 0, 2021-05-12T12:04:19.270 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd" 2021-05-12T12:04:19.270 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.270 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.271 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:04:19.271 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "host" 2021-05-12T12:04:19.271 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.271 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.271 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 2, 2021-05-12T12:04:19.272 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "chassis" 2021-05-12T12:04:19.272 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.272 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.272 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 3, 2021-05-12T12:04:19.273 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "rack" 2021-05-12T12:04:19.273 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.273 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.273 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 4, 2021-05-12T12:04:19.273 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "row" 2021-05-12T12:04:19.274 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.274 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.274 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 5, 2021-05-12T12:04:19.274 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pdu" 2021-05-12T12:04:19.274 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.275 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.275 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 6, 2021-05-12T12:04:19.275 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pod" 2021-05-12T12:04:19.275 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.275 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.276 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 7, 2021-05-12T12:04:19.276 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "room" 2021-05-12T12:04:19.276 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.276 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.277 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 8, 2021-05-12T12:04:19.277 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "datacenter" 2021-05-12T12:04:19.277 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.277 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.277 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 9, 2021-05-12T12:04:19.278 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "zone" 2021-05-12T12:04:19.278 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.278 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.278 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 10, 2021-05-12T12:04:19.278 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "region" 2021-05-12T12:04:19.279 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.279 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.279 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:04:19.279 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "root" 2021-05-12T12:04:19.279 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.280 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.280 INFO:tasks.workunit.client.0.smithi191.stdout: "buckets": [ 2021-05-12T12:04:19.280 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.280 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -1, 2021-05-12T12:04:19.281 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default", 2021-05-12T12:04:19.281 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:04:19.281 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:04:19.281 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:04:19.281 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:04:19.282 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:04:19.282 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:04:19.282 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.282 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:04:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:04:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:04:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.284 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.284 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -2, 2021-05-12T12:04:19.284 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default~hdd", 2021-05-12T12:04:19.284 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:04:19.284 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:04:19.285 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:04:19.285 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:04:19.285 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:04:19.285 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:04:19.285 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.286 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:04:19.286 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:04:19.286 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:04:19.286 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.287 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.287 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.287 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.287 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:04:19.287 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191", 2021-05-12T12:04:19.288 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:04:19.288 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:04:19.288 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:04:19.288 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:04:19.288 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:04:19.289 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:04:19.289 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.289 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:04:19.289 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:04:19.289 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:04:19.290 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.290 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.290 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:04:19.290 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:04:19.290 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:04:19.291 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.291 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.291 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:04:19.291 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:04:19.291 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:04:19.292 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.292 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.292 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.292 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.292 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:04:19.293 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191~hdd", 2021-05-12T12:04:19.293 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:04:19.293 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:04:19.293 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:04:19.294 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:04:19.294 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:04:19.294 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:04:19.294 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.294 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:04:19.295 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:04:19.295 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:04:19.295 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.295 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.295 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:04:19.296 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:04:19.296 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:04:19.296 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.296 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.297 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:04:19.297 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:04:19.297 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:04:19.297 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.297 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.298 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.298 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.298 INFO:tasks.workunit.client.0.smithi191.stdout: "rules": [ 2021-05-12T12:04:19.298 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.298 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 0, 2021-05-12T12:04:19.299 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "replicated_rule", 2021-05-12T12:04:19.299 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 0, 2021-05-12T12:04:19.299 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:04:19.299 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 1, 2021-05-12T12:04:19.300 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 10, 2021-05-12T12:04:19.300 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2021-05-12T12:04:19.300 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.300 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2021-05-12T12:04:19.300 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2021-05-12T12:04:19.301 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2021-05-12T12:04:19.301 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.301 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.301 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_firstn", 2021-05-12T12:04:19.301 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2021-05-12T12:04:19.302 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2021-05-12T12:04:19.302 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.302 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.302 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2021-05-12T12:04:19.303 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.303 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:04:19.303 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.303 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.303 INFO:tasks.workunit.client.0.smithi191.stdout: "tunables": { 2021-05-12T12:04:19.304 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_tries": 0, 2021-05-12T12:04:19.304 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:04:19.304 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_total_tries": 50, 2021-05-12T12:04:19.304 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:04:19.304 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:04:19.305 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_stable": 1, 2021-05-12T12:04:19.305 INFO:tasks.workunit.client.0.smithi191.stdout: "straw_calc_version": 1, 2021-05-12T12:04:19.305 INFO:tasks.workunit.client.0.smithi191.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:04:19.305 INFO:tasks.workunit.client.0.smithi191.stdout: "profile": "jewel", 2021-05-12T12:04:19.305 INFO:tasks.workunit.client.0.smithi191.stdout: "optimal_tunables": 1, 2021-05-12T12:04:19.306 INFO:tasks.workunit.client.0.smithi191.stdout: "legacy_tunables": 0, 2021-05-12T12:04:19.306 INFO:tasks.workunit.client.0.smithi191.stdout: "minimum_required_version": "jewel", 2021-05-12T12:04:19.306 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables": 1, 2021-05-12T12:04:19.306 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables2": 1, 2021-05-12T12:04:19.307 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v2_rules": 0, 2021-05-12T12:04:19.307 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables3": 1, 2021-05-12T12:04:19.307 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v3_rules": 0, 2021-05-12T12:04:19.307 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v4_buckets": 1, 2021-05-12T12:04:19.307 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables5": 1, 2021-05-12T12:04:19.308 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v5_rules": 0 2021-05-12T12:04:19.308 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.308 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_args": {} 2021-05-12T12:04:19.308 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.309 INFO:tasks.workunit.client.0.smithi191.stdout: "fsmap": { 2021-05-12T12:04:19.309 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:04:19.309 INFO:tasks.workunit.client.0.smithi191.stdout: "default_fscid": -1, 2021-05-12T12:04:19.309 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": { 2021-05-12T12:04:19.309 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": {}, 2021-05-12T12:04:19.310 INFO:tasks.workunit.client.0.smithi191.stdout: "ro_compat": {}, 2021-05-12T12:04:19.310 INFO:tasks.workunit.client.0.smithi191.stdout: "incompat": { 2021-05-12T12:04:19.310 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_1": "base v0.20", 2021-05-12T12:04:19.310 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:04:19.311 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:04:19.311 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:04:19.311 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:04:19.311 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:04:19.311 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_8": "no anchor table", 2021-05-12T12:04:19.312 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_9": "file layout v2", 2021-05-12T12:04:19.312 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:04:19.312 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.312 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.313 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_flags": { 2021-05-12T12:04:19.313 INFO:tasks.workunit.client.0.smithi191.stdout: "enable_multiple": true, 2021-05-12T12:04:19.313 INFO:tasks.workunit.client.0.smithi191.stdout: "ever_enabled_multiple": true 2021-05-12T12:04:19.313 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.313 INFO:tasks.workunit.client.0.smithi191.stdout: "standbys": [], 2021-05-12T12:04:19.314 INFO:tasks.workunit.client.0.smithi191.stdout: "filesystems": [] 2021-05-12T12:04:19.314 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.314 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:04:19.314 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:04:19.314 INFO:tasks.workunit.client.0.smithi191.stdout: "auth": { 2021-05-12T12:04:19.315 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:04:19.315 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 5, 2021-05-12T12:04:19.315 INFO:tasks.workunit.client.0.smithi191.stdout: "num_secrets": 3 2021-05-12T12:04:19.315 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.315 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:04:19.316 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_active": 1, 2021-05-12T12:04:19.316 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_unknown": 0, 2021-05-12T12:04:19.316 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osd": 3, 2021-05-12T12:04:19.316 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_sum": { 2021-05-12T12:04:19.317 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:04:19.317 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 47947, 2021-05-12T12:04:19.317 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:04:19.317 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:04:19.317 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:04:19.318 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:04:19.318 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:04:19.318 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:04:19.318 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:04:19.318 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:04:19.319 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:04:19.319 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:04:19.319 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:04:19.319 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:04:19.320 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 101, 2021-05-12T12:04:19.320 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 101, 2021-05-12T12:04:19.320 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:04:19.320 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:04:19.320 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:04:19.321 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:04:19.321 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:04:19.321 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:04:19.321 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:04:19.322 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:04:19.322 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:04:19.322 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:04:19.322 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:04:19.322 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:04:19.323 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:04:19.323 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:04:19.323 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:04:19.323 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:04:19.323 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:04:19.324 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:04:19.324 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:04:19.324 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:04:19.324 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:04:19.325 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:04:19.325 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:04:19.325 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:04:19.325 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:04:19.325 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.326 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:04:19.326 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:04:19.326 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:04:19.326 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:04:19.327 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 0, 2021-05-12T12:04:19.327 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 0, 2021-05-12T12:04:19.327 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:04:19.327 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:19.327 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:04:19.328 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:04:19.328 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:04:19.328 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.328 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 17, 2021-05-12T12:04:19.328 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 17, 2021-05-12T12:04:19.329 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 2, 2021-05-12T12:04:19.329 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 2, 2021-05-12T12:04:19.329 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 0 2021-05-12T12:04:19.329 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.329 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum": { 2021-05-12T12:04:19.330 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:04:19.330 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:04:19.330 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:04:19.330 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:04:19.330 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:04:19.331 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:04:19.331 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:04:19.331 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 18408, 2021-05-12T12:04:19.331 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2056, 2021-05-12T12:04:19.332 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:04:19.332 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16317, 2021-05-12T12:04:19.332 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314554392, 2021-05-12T12:04:19.332 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:04:19.332 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:04:19.333 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322103697408, 2021-05-12T12:04:19.333 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:04:19.333 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2105344, 2021-05-12T12:04:19.333 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 403479, 2021-05-12T12:04:19.333 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:04:19.334 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:19.334 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:04:19.334 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:04:19.334 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 16709520 2021-05-12T12:04:19.335 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.335 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:04:19.335 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:04:19.335 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:04:19.335 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:04:19.336 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:04:19.336 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:04:19.336 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:04:19.336 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.336 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:04:19.337 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 0, 2021-05-12T12:04:19.337 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 0, 2021-05-12T12:04:19.337 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 0, 2021-05-12T12:04:19.337 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 0 2021-05-12T12:04:19.337 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.338 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:04:19.338 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:04:19.338 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.338 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum_by_class": { 2021-05-12T12:04:19.339 INFO:tasks.workunit.client.0.smithi191.stdout: "hdd": { 2021-05-12T12:04:19.339 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:04:19.339 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:04:19.339 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:04:19.339 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:04:19.340 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:04:19.340 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:04:19.340 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:04:19.340 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 18408, 2021-05-12T12:04:19.341 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2056, 2021-05-12T12:04:19.341 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:04:19.341 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16317, 2021-05-12T12:04:19.341 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314554392, 2021-05-12T12:04:19.341 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:04:19.342 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:04:19.342 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322103697408, 2021-05-12T12:04:19.342 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:04:19.342 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2105344, 2021-05-12T12:04:19.342 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 403479, 2021-05-12T12:04:19.343 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:04:19.343 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:19.343 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:04:19.343 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:04:19.344 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 16709520 2021-05-12T12:04:19.344 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.344 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:04:19.344 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:04:19.344 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:04:19.345 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:04:19.345 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:04:19.345 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:04:19.345 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:04:19.345 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.346 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:04:19.346 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 0, 2021-05-12T12:04:19.346 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 0, 2021-05-12T12:04:19.346 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 0, 2021-05-12T12:04:19.347 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 0 2021-05-12T12:04:19.347 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.347 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:04:19.347 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:04:19.347 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.348 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.348 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_stats": [ 2021-05-12T12:04:19.348 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.348 INFO:tasks.workunit.client.0.smithi191.stdout: "poolid": 1, 2021-05-12T12:04:19.348 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:04:19.349 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:04:19.349 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 47947, 2021-05-12T12:04:19.349 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:04:19.349 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:04:19.349 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:04:19.350 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:04:19.350 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:04:19.350 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:04:19.350 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:04:19.350 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:04:19.351 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:04:19.351 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:04:19.351 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:04:19.351 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:04:19.351 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 101, 2021-05-12T12:04:19.352 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 101, 2021-05-12T12:04:19.352 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:04:19.352 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:04:19.352 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:04:19.352 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:04:19.353 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:04:19.353 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:04:19.353 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:04:19.353 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:04:19.353 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:04:19.354 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:04:19.354 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:04:19.354 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:04:19.354 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:04:19.355 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:04:19.355 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:04:19.355 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:04:19.355 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:04:19.355 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:04:19.356 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:04:19.356 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:04:19.356 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:04:19.356 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:04:19.356 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:04:19.357 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:04:19.357 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:04:19.357 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.357 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:04:19.358 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:04:19.358 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:04:19.358 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:04:19.358 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 1220608, 2021-05-12T12:04:19.358 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 142522, 2021-05-12T12:04:19.359 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:04:19.359 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:19.359 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:04:19.359 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:04:19.360 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:04:19.360 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.360 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 17, 2021-05-12T12:04:19.360 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 17, 2021-05-12T12:04:19.360 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 2, 2021-05-12T12:04:19.361 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 2, 2021-05-12T12:04:19.361 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 3 2021-05-12T12:04:19.361 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.361 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.361 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_stats": [ 2021-05-12T12:04:19.362 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.362 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:04:19.362 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 146028888084 2021-05-12T12:04:19.362 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.363 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.363 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:04:19.363 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 42949672977 2021-05-12T12:04:19.363 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.363 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.364 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:04:19.364 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 154618822675 2021-05-12T12:04:19.364 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.364 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.364 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_state": [ 2021-05-12T12:04:19.365 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.365 INFO:tasks.workunit.client.0.smithi191.stdout: "state": "active+undersized+degraded", 2021-05-12T12:04:19.365 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 1 2021-05-12T12:04:19.365 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.365 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.366 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_osd": [ 2021-05-12T12:04:19.366 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.366 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:04:19.366 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:04:19.367 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:04:19.367 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:04:19.367 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.367 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.367 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:04:19.368 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:04:19.368 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 0, 2021-05-12T12:04:19.368 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:04:19.368 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.368 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.369 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:04:19.369 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 1, 2021-05-12T12:04:19.369 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:04:19.369 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:04:19.369 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.370 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.370 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": [ 2021-05-12T12:04:19.370 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:04:19.370 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:04:19.371 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": {} 2021-05-12T12:04:19.371 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.371 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:04:19.371 INFO:tasks.workunit.client.0.smithi191.stdout: "servicemap": { 2021-05-12T12:04:19.371 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 6, 2021-05-12T12:04:19.372 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:04:05.448838+0000", 2021-05-12T12:04:19.372 INFO:tasks.workunit.client.0.smithi191.stdout: "services": {} 2021-05-12T12:04:19.372 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:04:19.372 INFO:tasks.workunit.client.0.smithi191.stdout: "paxos": { 2021-05-12T12:04:19.373 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:04:19.373 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 456, 2021-05-12T12:04:19.373 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pn": 0, 2021-05-12T12:04:19.373 INFO:tasks.workunit.client.0.smithi191.stdout: "accepted_pn": 0 2021-05-12T12:04:19.373 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:04:19.374 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:04:19.374 INFO:tasks.workunit.client.0.smithi191.stderr:report 1357001348 2021-05-12T12:04:19.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:04:19.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2021-05-12T12:04:19.375 INFO:tasks.workunit.client.0.smithi191.stdout:delay recovery 2021-05-12T12:04:19.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:19.473 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:19.474 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:03:07.997934+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:04:19.474 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:04:19.474 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:19.475 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:04:19.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:04:19.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2021-05-12T12:04:19.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:04:19.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:04:19.487 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:04:19.487 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:04:19.487 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:04:19.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.0.asok 2021-05-12T12:04:19.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:04:19.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T12:04:19.582 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:04:19.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:04:19.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2021-05-12T12:04:19.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:04:19.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:04:19.592 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:04:19.592 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:04:19.593 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:04:19.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.2.asok 2021-05-12T12:04:19.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:04:19.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T12:04:19.686 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:04:19.694 INFO:tasks.workunit.client.0.smithi191.stdout:revive divergent 1 2021-05-12T12:04:19.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2021-05-12T12:04:19.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:20.036 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:20.037 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:03:07.997934+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:04:20.037 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:04:20.037 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:20.039 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:04:20.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2021-05-12T12:04:20.794 INFO:tasks.workunit.client.0.smithi191.stderr:noup is set 2021-05-12T12:04:20.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2021-05-12T12:04:20.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:04:20.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:04:20.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:04:20.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:04:20.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:04:20.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:04:20.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:04:20.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:04:20.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:04:20.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:04:20.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:04:20.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:04:20.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:04:20.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:04:20.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:04:20.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:04:20.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:04:20.810 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:04:20.810 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:04:20.811 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:04:20.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:04:20.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:04:20.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:04:20.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:04:20.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:04:20.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:04:20.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:04:20.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:04:20.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:04:20.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:04:20.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:04:20.813 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:04:20.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:04:20.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:04:20.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:04:20.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:04:20.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:04:20.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:04:20.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:04:20.833 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:20.831+0000 7f79eaa650c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:04:20.849 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:20.846+0000 7f79eaa650c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:04:20.850 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:20.848+0000 7f79eaa650c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:04:21.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T12:04:21.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2021-05-12T12:04:21.366 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:21.363+0000 7f79eaa650c0 -1 Falling back to public interface 2021-05-12T12:04:24.262 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:24.259+0000 7f79eaa650c0 -1 osd.1 27 log_to_monitors {default=true} 2021-05-12T12:04:26.177 INFO:tasks.workunit.client.0.smithi191.stdout:delay recovery divergent 2021-05-12T12:04:26.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2021-05-12T12:04:26.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:26.533 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:26.533 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:03:07.997934+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:04:26.533 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:04:26.533 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:26.534 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:04:26.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2021-05-12T12:04:26.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:04:26.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:04:26.545 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:04:26.545 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:04:26.546 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:04:26.546 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.1.asok 2021-05-12T12:04:26.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:04:26.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T12:04:26.640 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:04:26.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2021-05-12T12:04:27.621 INFO:tasks.workunit.client.0.smithi191.stderr:noup is unset 2021-05-12T12:04:27.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2021-05-12T12:04:27.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:04:27.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:04:27.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:04:27.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:04:27.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:04:27.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:04:27.632 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:04:27.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:04:27.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:04:28.037 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 34 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/53991,v1:127.0.0.1:6803/53991] [v2:127.0.0.1:6804/53991,v1:127.0.0.1:6805/53991] exists,up 680f25cc-1994-4806-b5f0-2bb987fe27cd 2021-05-12T12:04:28.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:04:28.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:04:28.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:04:28.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2021-05-12T12:04:28.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:04:28.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:04:28.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:04:28.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:04:28.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:04:28.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:04:28.041 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:04:28.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:04:28.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:04:28.418 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/57264,v1:127.0.0.1:6819/57264] [v2:127.0.0.1:6820/57264,v1:127.0.0.1:6821/57264] exists,up 8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a 2021-05-12T12:04:28.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:04:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:04:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:04:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2021-05-12T12:04:28.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:04:28.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:04:28.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:04:28.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:04:28.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:04:28.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:04:28.422 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:04:28.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:04:28.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:04:28.806 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/54548,v1:127.0.0.1:6811/54548] [v2:127.0.0.1:6812/54548,v1:127.0.0.1:6813/54548] exists,up 9dbbea6e-ca5b-4917-9e87-b424e890b08e 2021-05-12T12:04:28.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:04:28.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:04:28.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:04:28.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:29.185 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:29.185 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:03:07.997934+0000 39'101 41:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:04:29.186 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:04:29.186 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:29.187 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:04:29.196 INFO:tasks.workunit.client.0.smithi191.stdout:wait for peering 2021-05-12T12:04:29.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2021-05-12T12:04:29.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:29.538 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:29.538 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-05-12T12:04:28.123277+0000 39'101 46:127 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:04:29.538 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:04:29.538 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:29.539 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:04:29.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2021-05-12T12:04:36.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2021-05-12T12:04:36.095 INFO:tasks.workunit.client.0.smithi191.stdout:killing divergent 1 2021-05-12T12:04:36.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:36.452 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:36.452 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-05-12T12:04:28.123277+0000 39'101 46:127 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:04:36.452 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:04:36.452 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:36.453 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:04:36.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2021-05-12T12:04:36.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:04:36.464 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:04:36.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:04:36.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:04:36.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:04:36.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:04:36.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.34183.out 2021-05-12T12:04:36.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.34183.out 2021-05-12T12:04:36.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:04:36.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:04:36.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:04:36.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:04:36.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:04:36.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.34183.out 2021-05-12T12:04:37.488 INFO:tasks.workunit.client.0.smithi191.stderr:Exporting 1.0 info 1.0( v 46'102 lc 0'0 (27'90,46'102] local-lis/les=44/46 n=101 ec=21/21 lis/c=44/21 les/c/f=46/22/0 sis=44 pruub=7.877776146s) 2021-05-12T12:04:37.495 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:023dd3d2:::existing_12:head# 2021-05-12T12:04:37.528 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:04c40a0c:::existing_93:head# 2021-05-12T12:04:37.546 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:064ff5fb:::existing_36:head# 2021-05-12T12:04:37.559 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:091f9f1d:::existing_94:head# 2021-05-12T12:04:37.571 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:0a865a93:::existing_54:head# 2021-05-12T12:04:37.572 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:0fed513c:::existing_41:head# 2021-05-12T12:04:37.584 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:13b9e0b8:::existing_92:head# 2021-05-12T12:04:37.604 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:169157fa:::existing_81:head# 2021-05-12T12:04:37.612 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:17731382:::existing_58:head# 2021-05-12T12:04:37.612 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:18a45a39:::existing_43:head# 2021-05-12T12:04:37.613 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:1c160f01:::existing_66:head# 2021-05-12T12:04:37.613 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:1dd6c249:::existing_97:head# 2021-05-12T12:04:37.614 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:227e0c77:::existing_7:head# 2021-05-12T12:04:37.615 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:23e1b063:::existing_59:head# 2021-05-12T12:04:37.615 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:28566edb:::existing_78:head# 2021-05-12T12:04:37.616 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:2a101182:::existing_20:head# 2021-05-12T12:04:37.616 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:2d1519da:::existing_96:head# 2021-05-12T12:04:37.617 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:3078b87a:::existing_86:head# 2021-05-12T12:04:37.617 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:30bde01b:::existing_15:head# 2021-05-12T12:04:37.618 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:317a7e19:::existing_56:head# 2021-05-12T12:04:37.618 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:38b9c7fa:::existing_33:head# 2021-05-12T12:04:37.626 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:3e16d84b:::existing_48:head# 2021-05-12T12:04:37.627 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:45fe8203:::existing_75:head# 2021-05-12T12:04:37.628 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:4e82c9ba:::existing_50:head# 2021-05-12T12:04:37.629 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:5298dcdb:::existing_57:head# 2021-05-12T12:04:37.630 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:52ebaa44:::existing_32:head# 2021-05-12T12:04:37.630 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:53588912:::existing_9:head# 2021-05-12T12:04:37.631 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:5b7e2de6:::existing_84:head# 2021-05-12T12:04:37.632 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:5da0255c:::existing_13:head# 2021-05-12T12:04:37.632 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:5da2dab8:::existing_53:head# 2021-05-12T12:04:37.633 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:5ee3514b:::existing_30:head# 2021-05-12T12:04:37.633 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:602f83fe:::foo:head# 2021-05-12T12:04:37.634 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:60426cc7:::existing_62:head# 2021-05-12T12:04:37.634 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:628918ca:::existing_79:head# 2021-05-12T12:04:37.635 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:63f8ab2d:::existing_77:head# 2021-05-12T12:04:37.635 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:67e1780f:::existing_27:head# 2021-05-12T12:04:37.636 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:68143ca1:::existing_71:head# 2021-05-12T12:04:37.636 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:6bac9509:::existing_29:head# 2021-05-12T12:04:37.637 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:6bbec798:::existing_45:head# 2021-05-12T12:04:37.637 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:6cb23a40:::existing_65:head# 2021-05-12T12:04:37.638 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:7055ec5e:::existing_83:head# 2021-05-12T12:04:37.639 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:70a54e2b:::existing_87:head# 2021-05-12T12:04:37.639 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:750ace21:::existing_19:head# 2021-05-12T12:04:37.640 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:78623b1d:::existing_6:head# 2021-05-12T12:04:37.640 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:79881152:::existing_63:head# 2021-05-12T12:04:37.641 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:7a80c0d5:::existing_99:head# 2021-05-12T12:04:37.641 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:81490677:::existing_76:head# 2021-05-12T12:04:37.642 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:85ec8e5d:::existing_85:head# 2021-05-12T12:04:37.642 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:8e2abbf0:::existing_49:head# 2021-05-12T12:04:37.643 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:8ee0fe0f:::existing_52:head# 2021-05-12T12:04:37.643 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:923c5d4d:::existing_5:head# 2021-05-12T12:04:37.644 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:92731e71:::existing_89:head# 2021-05-12T12:04:37.644 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:928205cf:::existing_82:head# 2021-05-12T12:04:37.645 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:93be2267:::existing_24:head# 2021-05-12T12:04:37.646 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:99ceda2c:::existing_44:head# 2021-05-12T12:04:37.646 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:9dd917ee:::existing_26:head# 2021-05-12T12:04:37.647 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:a28805a6:::existing_72:head# 2021-05-12T12:04:37.647 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:a294a8f6:::existing_16:head# 2021-05-12T12:04:37.648 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:a2f5473a:::existing_42:head# 2021-05-12T12:04:37.649 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:a504a533:::existing_90:head# 2021-05-12T12:04:37.649 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:a691337e:::existing_61:head# 2021-05-12T12:04:37.650 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:a6e8d7ac:::existing_28:head# 2021-05-12T12:04:37.650 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:a9fde86e:::existing_18:head# 2021-05-12T12:04:37.651 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:aa309faa:::existing_68:head# 2021-05-12T12:04:37.651 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:ae40b0ed:::existing_31:head# 2021-05-12T12:04:37.652 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:b279b1fe:::existing_10:head# 2021-05-12T12:04:37.653 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:b343e277:::existing_88:head# 2021-05-12T12:04:37.653 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:b7130e90:::existing_80:head# 2021-05-12T12:04:37.654 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:bc6f7309:::existing_70:head# 2021-05-12T12:04:37.654 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:be8489b7:::existing_38:head# 2021-05-12T12:04:37.655 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:beb4e5e3:::existing_47:head# 2021-05-12T12:04:37.656 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:bebec877:::existing_74:head# 2021-05-12T12:04:37.656 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:c04347d8:::existing_60:head# 2021-05-12T12:04:37.656 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:c156033e:::existing_100:head# 2021-05-12T12:04:37.656 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:c19f5a8e:::existing_8:head# 2021-05-12T12:04:37.657 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:c3e6ebe4:::existing_17:head# 2021-05-12T12:04:37.658 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:c6c5a7c2:::existing_39:head# 2021-05-12T12:04:37.658 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:cc1d0abf:::existing_95:head# 2021-05-12T12:04:37.659 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:ccb3865b:::existing_73:head# 2021-05-12T12:04:37.659 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:cd84e19e:::existing_40:head# 2021-05-12T12:04:37.659 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:d2cd64ce:::existing_37:head# 2021-05-12T12:04:37.660 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:d41a25c9:::existing_22:head# 2021-05-12T12:04:37.660 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:d7f088da:::existing_91:head# 2021-05-12T12:04:37.661 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:d9cbd1d6:::existing_64:head# 2021-05-12T12:04:37.661 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:dcc4ff82:::existing_98:head# 2021-05-12T12:04:37.661 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:dd677bd3:::existing_35:head# 2021-05-12T12:04:37.662 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:dd6d90e7:::existing_46:head# 2021-05-12T12:04:37.662 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:e0e22077:::existing_21:head# 2021-05-12T12:04:37.663 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:e255b710:::existing_55:head# 2021-05-12T12:04:37.664 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:eff1d5db:::existing_14:head# 2021-05-12T12:04:37.664 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:f1b156bf:::existing_23:head# 2021-05-12T12:04:37.665 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:f5ce4199:::existing_34:head# 2021-05-12T12:04:37.665 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:f6809456:::existing_11:head# 2021-05-12T12:04:37.666 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:f699a2a9:::existing_67:head# 2021-05-12T12:04:37.667 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:f9baac43:::existing_25:head# 2021-05-12T12:04:37.667 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:fb76688e:::existing_51:head# 2021-05-12T12:04:37.668 INFO:tasks.workunit.client.0.smithi191.stderr:Read #1:fbb92eec:::existing_69:head# 2021-05-12T12:04:37.668 INFO:tasks.workunit.client.0.smithi191.stdout: marking collection for removal 2021-05-12T12:04:37.669 INFO:tasks.workunit.client.0.smithi191.stderr:Export successful 2021-05-12T12:04:37.669 INFO:tasks.workunit.client.0.smithi191.stdout:setting '_remove' omap key 2021-05-12T12:04:37.670 INFO:tasks.workunit.client.0.smithi191.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-05-12T12:04:37.975 INFO:tasks.workunit.client.0.smithi191.stderr:Remove successful 2021-05-12T12:04:38.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.34183.out 2021-05-12T12:04:38.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:04:38.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:04:38.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:04:38.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:04:38.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:04:38.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.34183.out 2021-05-12T12:04:39.158 INFO:tasks.workunit.client.0.smithi191.stdout:get_pg_num_history pg_num_history pg_num_history(e46 pg_nums {1={21=1}} deleted_pools ) 2021-05-12T12:04:39.159 INFO:tasks.workunit.client.0.smithi191.stdout:Importing pgid 1.0 2021-05-12T12:04:39.159 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:023dd3d2:::existing_12:head# 2021-05-12T12:04:39.159 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.243 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:04c40a0c:::existing_93:head# 2021-05-12T12:04:39.243 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.309 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:064ff5fb:::existing_36:head# 2021-05-12T12:04:39.310 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.343 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:091f9f1d:::existing_94:head# 2021-05-12T12:04:39.343 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.376 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:0a865a93:::existing_54:head# 2021-05-12T12:04:39.377 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.410 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:0fed513c:::existing_41:head# 2021-05-12T12:04:39.410 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.444 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:13b9e0b8:::existing_92:head# 2021-05-12T12:04:39.444 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.477 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:169157fa:::existing_81:head# 2021-05-12T12:04:39.477 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.511 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:17731382:::existing_58:head# 2021-05-12T12:04:39.511 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.593 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:18a45a39:::existing_43:head# 2021-05-12T12:04:39.593 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.636 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:1c160f01:::existing_66:head# 2021-05-12T12:04:39.637 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.670 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:1dd6c249:::existing_97:head# 2021-05-12T12:04:39.670 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.703 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:227e0c77:::existing_7:head# 2021-05-12T12:04:39.704 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.737 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:23e1b063:::existing_59:head# 2021-05-12T12:04:39.737 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.771 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:28566edb:::existing_78:head# 2021-05-12T12:04:39.771 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.804 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:2a101182:::existing_20:head# 2021-05-12T12:04:39.804 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.838 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:2d1519da:::existing_96:head# 2021-05-12T12:04:39.838 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.871 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:3078b87a:::existing_86:head# 2021-05-12T12:04:39.871 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.938 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:30bde01b:::existing_15:head# 2021-05-12T12:04:39.938 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:39.975 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:317a7e19:::existing_56:head# 2021-05-12T12:04:39.975 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.030 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:38b9c7fa:::existing_33:head# 2021-05-12T12:04:40.030 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.064 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:3e16d84b:::existing_48:head# 2021-05-12T12:04:40.064 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.097 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:45fe8203:::existing_75:head# 2021-05-12T12:04:40.098 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.131 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:4e82c9ba:::existing_50:head# 2021-05-12T12:04:40.131 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.164 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:5298dcdb:::existing_57:head# 2021-05-12T12:04:40.165 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.228 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:52ebaa44:::existing_32:head# 2021-05-12T12:04:40.228 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.273 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:53588912:::existing_9:head# 2021-05-12T12:04:40.273 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.306 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:5b7e2de6:::existing_84:head# 2021-05-12T12:04:40.307 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.340 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:5da0255c:::existing_13:head# 2021-05-12T12:04:40.340 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.373 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:5da2dab8:::existing_53:head# 2021-05-12T12:04:40.374 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.399 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:5ee3514b:::existing_30:head# 2021-05-12T12:04:40.399 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.424 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:602f83fe:::foo:head# 2021-05-12T12:04:40.424 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.457 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:60426cc7:::existing_62:head# 2021-05-12T12:04:40.458 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.499 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:628918ca:::existing_79:head# 2021-05-12T12:04:40.500 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.575 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:63f8ab2d:::existing_77:head# 2021-05-12T12:04:40.575 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.608 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:67e1780f:::existing_27:head# 2021-05-12T12:04:40.609 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.642 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:68143ca1:::existing_71:head# 2021-05-12T12:04:40.642 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.675 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:6bac9509:::existing_29:head# 2021-05-12T12:04:40.676 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.700 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:6bbec798:::existing_45:head# 2021-05-12T12:04:40.701 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.726 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:6cb23a40:::existing_65:head# 2021-05-12T12:04:40.726 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.751 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:7055ec5e:::existing_83:head# 2021-05-12T12:04:40.751 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.789 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:70a54e2b:::existing_87:head# 2021-05-12T12:04:40.789 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:40.846 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:750ace21:::existing_19:head# 2021-05-12T12:04:41.093 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.095 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:78623b1d:::existing_6:head# 2021-05-12T12:04:41.095 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.095 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:79881152:::existing_63:head# 2021-05-12T12:04:41.096 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.096 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:7a80c0d5:::existing_99:head# 2021-05-12T12:04:41.096 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.096 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:81490677:::existing_76:head# 2021-05-12T12:04:41.097 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.097 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:85ec8e5d:::existing_85:head# 2021-05-12T12:04:41.097 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.097 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:8e2abbf0:::existing_49:head# 2021-05-12T12:04:41.097 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.098 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:8ee0fe0f:::existing_52:head# 2021-05-12T12:04:41.098 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.098 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:923c5d4d:::existing_5:head# 2021-05-12T12:04:41.098 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.116 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:92731e71:::existing_89:head# 2021-05-12T12:04:41.117 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.175 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:928205cf:::existing_82:head# 2021-05-12T12:04:41.176 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.200 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:93be2267:::existing_24:head# 2021-05-12T12:04:41.200 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.234 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:99ceda2c:::existing_44:head# 2021-05-12T12:04:41.235 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.268 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:9dd917ee:::existing_26:head# 2021-05-12T12:04:41.268 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.301 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:a28805a6:::existing_72:head# 2021-05-12T12:04:41.301 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.335 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:a294a8f6:::existing_16:head# 2021-05-12T12:04:41.335 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.368 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:a2f5473a:::existing_42:head# 2021-05-12T12:04:41.369 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.410 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:a504a533:::existing_90:head# 2021-05-12T12:04:41.410 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.502 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:a691337e:::existing_61:head# 2021-05-12T12:04:41.503 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.586 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:a6e8d7ac:::existing_28:head# 2021-05-12T12:04:41.586 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.653 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:a9fde86e:::existing_18:head# 2021-05-12T12:04:41.654 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.703 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:aa309faa:::existing_68:head# 2021-05-12T12:04:41.704 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.791 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:ae40b0ed:::existing_31:head# 2021-05-12T12:04:41.792 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.863 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:b279b1fe:::existing_10:head# 2021-05-12T12:04:41.863 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.913 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:b343e277:::existing_88:head# 2021-05-12T12:04:41.914 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:41.967 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:b7130e90:::existing_80:head# 2021-05-12T12:04:41.968 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.047 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:bc6f7309:::existing_70:head# 2021-05-12T12:04:42.048 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.131 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:be8489b7:::existing_38:head# 2021-05-12T12:04:42.132 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.181 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:beb4e5e3:::existing_47:head# 2021-05-12T12:04:42.182 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.232 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:bebec877:::existing_74:head# 2021-05-12T12:04:42.232 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.291 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:c04347d8:::existing_60:head# 2021-05-12T12:04:42.291 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.341 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:c156033e:::existing_100:head# 2021-05-12T12:04:42.341 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.418 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:c19f5a8e:::existing_8:head# 2021-05-12T12:04:42.418 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.467 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:c3e6ebe4:::existing_17:head# 2021-05-12T12:04:42.467 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.509 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:c6c5a7c2:::existing_39:head# 2021-05-12T12:04:42.509 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.551 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:cc1d0abf:::existing_95:head# 2021-05-12T12:04:42.551 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.584 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:ccb3865b:::existing_73:head# 2021-05-12T12:04:42.585 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.618 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:cd84e19e:::existing_40:head# 2021-05-12T12:04:42.618 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.660 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:d2cd64ce:::existing_37:head# 2021-05-12T12:04:42.661 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.739 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:d41a25c9:::existing_22:head# 2021-05-12T12:04:42.739 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.794 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:d7f088da:::existing_91:head# 2021-05-12T12:04:42.795 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.828 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:d9cbd1d6:::existing_64:head# 2021-05-12T12:04:42.828 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.878 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:dcc4ff82:::existing_98:head# 2021-05-12T12:04:42.879 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.928 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:dd677bd3:::existing_35:head# 2021-05-12T12:04:42.929 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:42.988 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:dd6d90e7:::existing_46:head# 2021-05-12T12:04:42.988 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.071 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:e0e22077:::existing_21:head# 2021-05-12T12:04:43.072 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.121 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:e255b710:::existing_55:head# 2021-05-12T12:04:43.122 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.172 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:eff1d5db:::existing_14:head# 2021-05-12T12:04:43.172 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.225 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:f1b156bf:::existing_23:head# 2021-05-12T12:04:43.226 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.323 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:f5ce4199:::existing_34:head# 2021-05-12T12:04:43.323 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.407 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:f6809456:::existing_11:head# 2021-05-12T12:04:43.407 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.457 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:f699a2a9:::existing_67:head# 2021-05-12T12:04:43.457 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.507 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:f9baac43:::existing_25:head# 2021-05-12T12:04:43.508 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.591 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:fb76688e:::existing_51:head# 2021-05-12T12:04:43.592 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.684 INFO:tasks.workunit.client.0.smithi191.stdout:Write #1:fbb92eec:::existing_69:head# 2021-05-12T12:04:43.684 INFO:tasks.workunit.client.0.smithi191.stdout:snapset 0=[]:{} 2021-05-12T12:04:43.708 INFO:tasks.workunit.client.0.smithi191.stdout:write_pg epoch 46 info 1.0( v 46'102 lc 0'0 (27'90,46'102] local-lis/les=44/46 n=101 ec=21/21 lis/c=44/21 les/c/f=46/22/0 sis=44 pruub=7.877776146s) 2021-05-12T12:04:43.750 INFO:tasks.workunit.client.0.smithi191.stdout:Import successful 2021-05-12T12:04:44.269 INFO:tasks.workunit.client.0.smithi191.stdout:reviving divergent 1 2021-05-12T12:04:44.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2021-05-12T12:04:44.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:44.620 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:44.621 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-05-12T12:04:28.123277+0000 39'101 46:127 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:04:44.621 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:04:44.621 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:44.622 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:04:44.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2021-05-12T12:04:44.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:04:44.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:04:44.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:04:44.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:04:44.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:04:44.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:04:44.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:04:44.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:04:44.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:04:44.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:04:44.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:04:44.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:04:44.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:04:44.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:04:44.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:04:44.640 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:04:44.640 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:04:44.640 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:04:44.640 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:04:44.641 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:04:44.641 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:04:44.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:04:44.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:04:44.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:04:44.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:04:44.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:04:44.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:04:44.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:04:44.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:04:44.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:04:44.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:04:44.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:04:44.644 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:04:44.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:04:44.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:04:44.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:04:44.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:04:44.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:04:44.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:04:44.662 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:44.659+0000 7fe097d280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:04:44.664 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:44.661+0000 7fe097d280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:04:44.665 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:44.662+0000 7fe097d280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:04:44.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:04:44.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:04:44.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:04:44.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:04:45.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:04:45.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:04:45.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:04:45.001 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:04:45.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:04:45.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:04:45.180 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:45.178+0000 7fe097d280c0 -1 Falling back to public interface 2021-05-12T12:04:45.390 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/57264,v1:127.0.0.1:6819/57264] [v2:127.0.0.1:6820/57264,v1:127.0.0.1:6821/57264] exists,up 8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a 2021-05-12T12:04:45.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:04:45.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:04:45.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:04:45.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2021-05-12T12:04:45.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:04:45.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:04:45.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:04:45.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:04:45.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:04:45.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:04:45.394 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:04:45.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:04:45.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:04:45.822 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/57264,v1:127.0.0.1:6819/57264] [v2:127.0.0.1:6820/57264,v1:127.0.0.1:6821/57264] exists,up 8f5a2e21-e8df-4abd-9a1b-87a4b1b9da5a 2021-05-12T12:04:45.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:04:45.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:04:45.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:04:45.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2021-05-12T12:04:47.899 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:04:47.896+0000 7fe097d280c0 -1 osd.1 46 log_to_monitors {default=true} 2021-05-12T12:05:05.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2021-05-12T12:05:05.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:05:05.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:05:05.826 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:05:05.826 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:05.826 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:05.826 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.1.asok 2021-05-12T12:05:05.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:05:05.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.1.asok dump_ops_in_flight 2021-05-12T12:05:05.927 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:05:05.927 INFO:tasks.workunit.client.0.smithi191.stdout: "ops": [], 2021-05-12T12:05:05.927 INFO:tasks.workunit.client.0.smithi191.stdout: "num_ops": 0 2021-05-12T12:05:05.928 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:05:05.935 INFO:tasks.workunit.client.0.smithi191.stdout:allowing recovery 2021-05-12T12:05:05.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2021-05-12T12:05:05.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:05:06.280 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:05:06.280 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+recovering 2021-05-12T12:05:01.828581+0000 46'102 50:147 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:05:06.281 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:05:06.281 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:05:06.282 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:05:06.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-05-12T12:05:06.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T12:05:06.417 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:05:06.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-05-12T12:05:06.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T12:05:06.549 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:05:06.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-05-12T12:05:06.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T12:05:06.681 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:05:06.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2021-05-12T12:05:06.690 INFO:tasks.workunit.client.0.smithi191.stdout:reading divergent objects 2021-05-12T12:05:06.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:05:07.033 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:05:07.033 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+recovering 2021-05-12T12:05:01.828581+0000 46'102 50:147 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:13.863726+0000 0'0 2021-05-12T12:02:13.863726+0000 0 2021-05-12T12:05:07.034 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:05:07.034 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:05:07.035 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:05:07.044 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2021-05-12T12:05:07.045 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2021-05-12T12:05:07.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:05:07.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T12:05:07.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:05:07.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T12:05:07.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:05:07.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T12:05:07.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:05:07.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T12:05:07.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2021-05-12T12:05:07.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:05:07.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2021-05-12T12:05:07.203 INFO:tasks.workunit.client.0.smithi191.stderr:error getting test/newobject_1: (2) No such file or directory 2021-05-12T12:05:07.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:05:07.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2021-05-12T12:05:07.233 INFO:tasks.workunit.client.0.smithi191.stderr:error getting test/newobject_2: (2) No such file or directory 2021-05-12T12:05:07.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2021-05-12T12:05:07.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-05-12T12:05:07.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:05:07.239 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 27'106 (27'4) delete 1:70b3bf12:::existing_4:head by client.4549.0:1 2021-05-12T12:02:44.397377+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:07.239 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: keeping 27'106 (27'4) delete 1:70b3bf12:::existing_4:head by client.4549.0:1 2021-05-12T12:02:44.397377+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:07.239 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:07.239 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 27'4 first_divergent_update: 27'106 last_divergent_update: 27'106 2021-05-12T12:05:07.240 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: more recent entry found: 39'101 (27'4) modify 1:70b3bf12:::existing_4:head by client.4664.0:1 2021-05-12T12:03:43.649256+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2021-05-12T12:05:07.240 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 27'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4542.0:1 2021-05-12T12:02:44.392200+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.240 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: keeping 27'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4542.0:1 2021-05-12T12:02:44.392200+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.240 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:07.241 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 27'104 last_divergent_update: 27'104 2021-05-12T12:05:07.241 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2021-05-12T12:05:07.241 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-05-12T12:05:07.241 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 27'105 (27'3) delete 1:8881e6d8:::existing_3:head by client.4548.0:1 2021-05-12T12:02:44.396493+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:07.242 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: keeping 27'105 (27'3) delete 1:8881e6d8:::existing_3:head by client.4548.0:1 2021-05-12T12:02:44.396493+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:07.242 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-05-12T12:05:07.243 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 27'3 first_divergent_update: 27'105 last_divergent_update: 27'105 2021-05-12T12:05:07.243 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2021-05-12T12:05:07.243 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:05:07.243 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 27'106 2021-05-12T12:05:07.244 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 27'105 (27'3) delete 1:8881e6d8:::existing_3:head by client.4548.0:1 2021-05-12T12:02:44.396493+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:07.244 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-05-12T12:05:07.244 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 27'3 <= info.log_tail 27'90 2021-05-12T12:05:07.244 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 27'101 (27'2) modify 1:97db2a0f:::existing_2:head by client.4536.0:1 2021-05-12T12:02:44.379750+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.245 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: keeping 27'101 (27'2) modify 1:97db2a0f:::existing_2:head by client.4536.0:1 2021-05-12T12:02:44.379750+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.245 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:07.245 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 27'2 first_divergent_update: 27'101 last_divergent_update: 27'101 2021-05-12T12:05:07.245 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2021-05-12T12:05:07.246 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:05:07.246 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 27'106 2021-05-12T12:05:07.246 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 27'101 (27'2) modify 1:97db2a0f:::existing_2:head by client.4536.0:1 2021-05-12T12:02:44.379750+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.246 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-05-12T12:05:07.247 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 27'2 <= info.log_tail 27'90 2021-05-12T12:05:07.247 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 27'102 (27'1) modify 1:d858c035:::existing_1:head by client.4537.0:1 2021-05-12T12:02:44.381679+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.247 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: keeping 27'102 (27'1) modify 1:d858c035:::existing_1:head by client.4537.0:1 2021-05-12T12:02:44.381679+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.247 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:07.248 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 27'1 first_divergent_update: 27'102 last_divergent_update: 27'102 2021-05-12T12:05:07.248 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2021-05-12T12:05:07.248 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:05:07.249 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 27'106 2021-05-12T12:05:07.249 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 27'102 (27'1) modify 1:d858c035:::existing_1:head by client.4537.0:1 2021-05-12T12:02:44.381679+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.249 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-05-12T12:05:07.249 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 27'1 <= info.log_tail 27'90 2021-05-12T12:05:07.249 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 27'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4543.0:1 2021-05-12T12:02:44.391960+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.250 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 20 _merge_object_divergent_entries: keeping 27'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4543.0:1 2021-05-12T12:02:44.391960+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:05:07.250 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:07.250 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 27'103 last_divergent_update: 27'103 2021-05-12T12:05:07.250 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2021-05-12T12:05:07.251 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:27.573+0000 7f79c695b700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-05-12T12:05:07.251 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-05-12T12:05:07.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:05:07.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-05-12T12:05:07.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:05:07.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2021-05-12T12:05:07.253 INFO:tasks.workunit.client.0.smithi191.stdout:success 2021-05-12T12:05:07.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.34183.out 2021-05-12T12:05:07.254 INFO:tasks.workunit.client.0.smithi191.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.34183.out': No such file or directory 2021-05-12T12:05:07.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2021-05-12T12:05:07.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T12:05:07.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T12:05:07.756 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test' does not exist 2021-05-12T12:05:07.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2021-05-12T12:05:07.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:05:07.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:05:07.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:05:07.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:05:07.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:05:07.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:05:07.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T12:05:07.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:05:07.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:05:07.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:05:07.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:05:07.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:05:07.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:05:07.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:05:07.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:05:07.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:05:07.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:05:07.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:05:07.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:05:07.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:05:07.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:05:07.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:05:07.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:05:07.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:05:07.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:05:07.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:05:07.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:05:07.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:05:07.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:05:07.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:05:07.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:07.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:07.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34183 2021-05-12T12:05:07.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:05:07.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:05:07.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T12:05:07.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T12:05:07.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T12:05:07.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T12:05:07.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:05:07.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:05:07.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:05:07.902 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:05:07.902 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:05:07.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:05:07.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:05:07.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:05:07.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:05:07.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:05:07.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:05:07.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:05:07.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:05:07.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:05:07.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:05:07.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:05:07.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:05:07.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:05:07.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:05:07.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:05:07.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:05:07.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:05:07.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:05:07.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:07.910 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:07.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34183 2021-05-12T12:05:07.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:05:07.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:05:07.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T12:05:07.912 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:05:07.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:07.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:07.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34183 2021-05-12T12:05:07.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:05:07.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:05:07.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:05:07.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T12:05:07.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2021-05-12T12:05:07.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2021-05-12T12:05:07.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2021-05-12T12:05:07.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2021-05-12T12:05:07.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2021-05-12T12:05:07.915 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2021-05-12T12:05:07.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2021-05-12T12:05:07.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2021-05-12T12:05:07.917 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:05:07.918 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:05:07.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2021-05-12T12:05:07.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T12:05:07.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:05:07.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:05:07.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:05:07.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T12:05:07.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T12:05:08.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:05:08.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:08.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:08.233 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:08.233 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:08.233 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:08.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:05:08.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:05:08.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:05:08.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:05:08.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:05:08.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:05:08.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:05:08.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:05:08.486 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:05:08.486 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:05:08.486 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:05:08.486 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:05:08.487 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:08.487 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:08.487 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-mon.a.asok 2021-05-12T12:05:08.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:05:08.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.34183/ceph-mon.a.asok config get fsid 2021-05-12T12:05:08.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:05:08.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:05:08.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:05:08.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:05:08.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:05:08.583 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:05:08.583 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:05:08.583 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:05:08.583 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:05:08.584 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:08.584 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:08.584 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-mon.a.asok 2021-05-12T12:05:08.584 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:05:08.584 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.34183/ceph-mon.a.asok config get mon_host 2021-05-12T12:05:08.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2021-05-12T12:05:08.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T12:05:08.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:05:08.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:05:08.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:05:08.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T12:05:08.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:05:09.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:05:09.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:09.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:09.554 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:09.554 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:09.554 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:09.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:05:09.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:05:09.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:05:09.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-05-12T12:05:09.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2021-05-12T12:05:09.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:05:09.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:05:09.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:05:09.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:05:09.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:05:09.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:05:09.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:05:09.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:05:09.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:05:09.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:05:09.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:05:09.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:05:09.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:05:09.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:05:09.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:05:09.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:09.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:09.593 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:09.593 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:09.593 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:09.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:05:09.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:05:09.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:05:09.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:05:09.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:05:09.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:05:09.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:05:09.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:05:09.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:05:09.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:09.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:05:09.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:05:09.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:05:09.596 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:05:09.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=efb617d0-a38f-4017-94d3-fc68ac2ed9be 2021-05-12T12:05:09.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 efb617d0-a38f-4017-94d3-fc68ac2ed9be' 2021-05-12T12:05:09.597 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 efb617d0-a38f-4017-94d3-fc68ac2ed9be 2021-05-12T12:05:09.597 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:05:09.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB1xJtgFZn8IxAA1JHR421NS5w3QAlKc11MTQ== 2021-05-12T12:05:09.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB1xJtgFZn8IxAA1JHR421NS5w3QAlKc11MTQ=="}' 2021-05-12T12:05:09.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new efb617d0-a38f-4017-94d3-fc68ac2ed9be -i td/divergent-priors/0/new.json 2021-05-12T12:05:09.934 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:05:09.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T12:05:09.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB1xJtgFZn8IxAA1JHR421NS5w3QAlKc11MTQ== --osd-uuid efb617d0-a38f-4017-94d3-fc68ac2ed9be 2021-05-12T12:05:09.976 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:09.972+0000 7f43e41c80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:09.977 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:09.975+0000 7f43e41c80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:09.979 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:09.977+0000 7f43e41c80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:09.980 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:09.977+0000 7f43e41c80c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:05:09.980 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:09.977+0000 7f43e41c80c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:05:09.981 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:09.977+0000 7f43e41c80c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:05:10.057 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:10.054+0000 7f43e41c80c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T12:05:14.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T12:05:14.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:05:14.898 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:05:14.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:05:14.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:05:15.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:05:15.386 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:05:15.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:05:15.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:15.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:05:15.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:05:15.413 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:15.411+0000 7ff598c640c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:15.415 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:15.413+0000 7ff598c640c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:15.417 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:15.414+0000 7ff598c640c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:15.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:05:15.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:05:15.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:05:15.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:05:15.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:05:15.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:15.756 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:05:15.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:05:15.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:15.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:05:15.933 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:15.931+0000 7ff598c640c0 -1 Falling back to public interface 2021-05-12T12:05:16.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:17.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:17.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:17.146 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:05:17.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:05:17.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:17.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:05:17.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:18.108 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:18.105+0000 7ff598c640c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:05:18.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:18.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:18.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:05:18.520 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:05:18.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:18.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:05:18.946 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/60738,v1:127.0.0.1:6803/60738] [v2:127.0.0.1:6804/60738,v1:127.0.0.1:6805/60738] exists,up efb617d0-a38f-4017-94d3-fc68ac2ed9be 2021-05-12T12:05:18.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:05:18.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:05:18.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:05:18.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-05-12T12:05:18.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2021-05-12T12:05:18.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:05:18.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:05:18.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:05:18.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:05:18.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:05:18.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:05:18.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:05:18.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:05:18.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:05:18.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:05:18.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:05:18.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:05:18.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:05:18.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:05:18.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:05:18.955 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:18.955 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:18.955 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:18.955 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:18.955 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:18.956 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:05:18.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:05:18.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:05:18.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:05:18.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:05:18.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:05:18.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:05:18.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:05:18.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:05:18.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:18.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:05:18.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:05:18.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:05:18.958 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:05:18.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=5fe20a5a-c3fb-42c5-aadd-2c4671a13114 2021-05-12T12:05:18.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 5fe20a5a-c3fb-42c5-aadd-2c4671a13114' 2021-05-12T12:05:18.959 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 5fe20a5a-c3fb-42c5-aadd-2c4671a13114 2021-05-12T12:05:18.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:05:18.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB+xJtg0W+TORAATSvIsYpATj0UhMluI4lhqA== 2021-05-12T12:05:18.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB+xJtg0W+TORAATSvIsYpATj0UhMluI4lhqA=="}' 2021-05-12T12:05:18.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 5fe20a5a-c3fb-42c5-aadd-2c4671a13114 -i td/divergent-priors/1/new.json 2021-05-12T12:05:19.544 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:05:19.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T12:05:19.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB+xJtg0W+TORAATSvIsYpATj0UhMluI4lhqA== --osd-uuid 5fe20a5a-c3fb-42c5-aadd-2c4671a13114 2021-05-12T12:05:19.586 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:19.584+0000 7f569435b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:19.588 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:19.586+0000 7f569435b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:19.589 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:19.587+0000 7f569435b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:19.590 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:19.588+0000 7f569435b0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:05:19.591 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:19.588+0000 7f569435b0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:05:19.591 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:19.588+0000 7f569435b0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:05:19.639 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:19.636+0000 7f569435b0c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T12:05:24.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T12:05:24.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:05:24.546 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:05:24.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:05:24.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:05:25.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:05:25.037 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:05:25.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:25.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:05:25.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:05:25.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:05:25.065 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:25.062+0000 7fd1f86130c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:25.086 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:25.083+0000 7fd1f86130c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:25.087 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:25.085+0000 7fd1f86130c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:25.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:05:25.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:05:25.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:05:25.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:05:25.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:05:25.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:25.406 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:05:25.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:05:25.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:25.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:05:25.603 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:25.600+0000 7fd1f86130c0 -1 Falling back to public interface 2021-05-12T12:05:25.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:26.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:26.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:26.800 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:05:26.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:05:26.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:26.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:05:27.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:27.830 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:27.828+0000 7fd1f86130c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:05:28.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:28.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:28.173 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:05:28.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:05:28.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:28.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:05:28.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:29.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:29.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:29.563 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:05:29.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:05:29.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:29.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:05:29.927 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/62131,v1:127.0.0.1:6811/62131] [v2:127.0.0.1:6812/62131,v1:127.0.0.1:6813/62131] exists,up 5fe20a5a-c3fb-42c5-aadd-2c4671a13114 2021-05-12T12:05:29.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:05:29.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:05:29.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:05:29.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-05-12T12:05:29.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2021-05-12T12:05:29.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:05:29.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:05:29.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:05:29.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:05:29.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:05:29.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:05:29.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:05:29.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:05:29.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:05:29.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:05:29.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:05:29.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:05:29.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:05:29.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:05:29.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:05:29.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:29.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:29.937 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:29.937 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:29.937 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:05:29.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:05:29.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:05:29.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:05:29.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:05:29.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:05:29.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:05:29.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:05:29.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:05:29.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:05:29.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:29.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:05:29.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:05:29.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:05:29.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:05:29.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=16b8b89a-4aa1-4296-88f5-d60bf3eabab1 2021-05-12T12:05:29.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 16b8b89a-4aa1-4296-88f5-d60bf3eabab1' 2021-05-12T12:05:29.941 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 16b8b89a-4aa1-4296-88f5-d60bf3eabab1 2021-05-12T12:05:29.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:05:29.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCJxJtgv4B8OBAAIp+eDMq4qd9ZCZDkeWfFZw== 2021-05-12T12:05:29.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCJxJtgv4B8OBAAIp+eDMq4qd9ZCZDkeWfFZw=="}' 2021-05-12T12:05:29.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 16b8b89a-4aa1-4296-88f5-d60bf3eabab1 -i td/divergent-priors/2/new.json 2021-05-12T12:05:30.436 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:05:30.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T12:05:30.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCJxJtgv4B8OBAAIp+eDMq4qd9ZCZDkeWfFZw== --osd-uuid 16b8b89a-4aa1-4296-88f5-d60bf3eabab1 2021-05-12T12:05:30.477 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:30.474+0000 7fc890aad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:30.479 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:30.476+0000 7fc890aad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:30.481 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:30.478+0000 7fc890aad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:30.481 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:30.478+0000 7fc890aad0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:05:30.482 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:30.478+0000 7fc890aad0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:05:30.482 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:30.478+0000 7fc890aad0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:05:30.527 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:30.525+0000 7fc890aad0c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T12:05:35.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T12:05:35.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:05:35.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:05:35.659 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:05:35.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:05:36.151 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:05:36.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:05:36.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:36.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:05:36.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:05:36.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:05:36.179 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:36.176+0000 7f6e074190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:36.181 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:36.178+0000 7f6e074190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:36.181 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:36.179+0000 7f6e074190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:36.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:05:36.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:05:36.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:05:36.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:05:36.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:05:36.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:36.524 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:05:36.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:05:36.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:36.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:05:36.697 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:36.695+0000 7f6e074190c0 -1 Falling back to public interface 2021-05-12T12:05:36.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:37.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:37.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:37.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:05:37.895 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:05:37.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:37.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:05:38.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:39.039 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:05:39.036+0000 7f6e074190c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:05:39.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:39.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:39.271 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:05:39.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:05:39.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:39.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:05:39.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:40.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:40.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:40.754 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:05:40.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:05:40.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:40.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:05:41.128 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63542,v1:127.0.0.1:6819/63542] [v2:127.0.0.1:6820/63542,v1:127.0.0.1:6821/63542] exists,up 16b8b89a-4aa1-4296-88f5-d60bf3eabab1 2021-05-12T12:05:41.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:05:41.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:05:41.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:05:41.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2021-05-12T12:05:41.898 INFO:tasks.workunit.client.0.smithi191.stderr:noout is set 2021-05-12T12:05:41.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2021-05-12T12:05:42.558 INFO:tasks.workunit.client.0.smithi191.stderr:noin is set 2021-05-12T12:05:42.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2021-05-12T12:05:43.360 INFO:tasks.workunit.client.0.smithi191.stderr:nodown is set 2021-05-12T12:05:43.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2021-05-12T12:05:43.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-05-12T12:05:43.986 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test' already exists 2021-05-12T12:05:43.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:05:44.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2021-05-12T12:05:45.784 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 size to 3 2021-05-12T12:05:45.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2021-05-12T12:05:46.518 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 min_size to 2 2021-05-12T12:05:46.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2021-05-12T12:05:47.247 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 pg_autoscale_mode to on 2021-05-12T12:05:47.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2021-05-12T12:05:47.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:05:47.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:05:47.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:05:47.625 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:05:47.625 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:05:47.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:05:47.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:05:47.626 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:05:47.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836488 2021-05-12T12:05:47.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836488 2021-05-12T12:05:47.759 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488' 2021-05-12T12:05:47.759 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:05:47.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:05:47.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672966 2021-05-12T12:05:47.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672966 2021-05-12T12:05:47.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2021-05-12T12:05:47.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:05:47.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:05:48.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509444 2021-05-12T12:05:48.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509444 2021-05-12T12:05:48.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509444' 2021-05-12T12:05:48.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:05:48.028 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2021-05-12T12:05:48.028 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:05:48.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:05:48.029 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836488 2021-05-12T12:05:48.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:05:48.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836488 2021-05-12T12:05:48.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-05-12T12:05:48.031 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836488 2021-05-12T12:05:48.031 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:05:48.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836488 2021-05-12T12:05:48.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:05:48.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-05-12T12:05:48.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:05:48.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:05:48.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672966 2021-05-12T12:05:48.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:05:48.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672966 2021-05-12T12:05:48.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-05-12T12:05:48.402 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672966 2021-05-12T12:05:48.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:05:48.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672966 2021-05-12T12:05:48.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:05:48.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-05-12T12:05:48.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:05:48.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:05:48.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509444 2021-05-12T12:05:48.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:05:48.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509444 2021-05-12T12:05:48.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-05-12T12:05:48.794 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 64424509444 2021-05-12T12:05:48.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:05:49.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509444 -lt 64424509444 2021-05-12T12:05:49.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2021-05-12T12:05:49.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:05:49.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:05:49.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:05:49.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:05:49.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:05:49.164 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:05:49.164 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:05:49.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:05:49.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:05:49.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:05:49.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:05:49.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:05:49.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:05:49.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:05:49.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:05:49.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:05:49.616 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:05:49.616 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:05:49.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:05:49.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:05:49.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:05:49.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836491 2021-05-12T12:05:49.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836491 2021-05-12T12:05:49.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491' 2021-05-12T12:05:49.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:05:49.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:05:49.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672969 2021-05-12T12:05:49.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672969 2021-05-12T12:05:49.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2021-05-12T12:05:49.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:05:49.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:05:50.018 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509446 2021-05-12T12:05:50.018 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509446 2021-05-12T12:05:50.018 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509446' 2021-05-12T12:05:50.018 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:05:50.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-05-12T12:05:50.019 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:05:50.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:05:50.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836491 2021-05-12T12:05:50.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:05:50.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836491 2021-05-12T12:05:50.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-05-12T12:05:50.021 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836491 2021-05-12T12:05:50.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:05:50.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836491 2021-05-12T12:05:50.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:05:50.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-05-12T12:05:50.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:05:50.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:05:50.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672969 2021-05-12T12:05:50.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:05:50.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672969 2021-05-12T12:05:50.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-05-12T12:05:50.421 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672969 2021-05-12T12:05:50.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:05:50.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672969 2021-05-12T12:05:50.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:05:50.802 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2021-05-12T12:05:50.802 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:05:50.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:05:50.803 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509446 2021-05-12T12:05:50.803 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:05:50.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509446 2021-05-12T12:05:50.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-05-12T12:05:50.805 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 64424509446 2021-05-12T12:05:50.805 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:05:51.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509446 -lt 64424509446 2021-05-12T12:05:51.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:05:51.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:05:51.171 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:05:51.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:05:51.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:05:51.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:05:51.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:05:51.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:05:51.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:05:51.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:05:51.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:05:52.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:05:52.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:05:52.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:05:52.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:05:52.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:05:52.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:05:52.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:05:52.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2021-05-12T12:05:52.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2021-05-12T12:05:52.830 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:05:52.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2021-05-12T12:05:52.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2021-05-12T12:05:52.840 INFO:tasks.workunit.client.0.smithi191.stdout:primary and soon to be divergent is 1 2021-05-12T12:05:52.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:05:53.186 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:05:53.186 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:05:45.077119+0000 0'0 29:13 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:05:53.187 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:05:53.187 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:05:53.188 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:05:53.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2021-05-12T12:05:53.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-05-12T12:05:53.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2021-05-12T12:05:53.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2021-05-12T12:05:53.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-05-12T12:05:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2021-05-12T12:05:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2021-05-12T12:05:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-05-12T12:05:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2021-05-12T12:05:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2021-05-12T12:05:53.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2021-05-12T12:05:53.201 INFO:tasks.workunit.client.0.smithi191.stdout:writing initial objects 2021-05-12T12:05:53.202 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2021-05-12T12:05:53.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:53.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2021-05-12T12:05:53.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:53.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2021-05-12T12:05:53.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:53.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2021-05-12T12:05:53.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:53.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2021-05-12T12:05:53.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:53.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2021-05-12T12:05:53.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:53.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2021-05-12T12:05:54.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:54.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2021-05-12T12:05:54.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:54.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2021-05-12T12:05:54.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:54.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2021-05-12T12:05:54.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:54.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2021-05-12T12:05:54.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:54.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2021-05-12T12:05:54.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:54.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2021-05-12T12:05:54.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:54.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2021-05-12T12:05:55.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:55.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2021-05-12T12:05:55.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:55.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2021-05-12T12:05:55.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:55.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2021-05-12T12:05:55.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:55.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2021-05-12T12:05:55.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:55.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2021-05-12T12:05:55.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:55.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2021-05-12T12:05:55.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:55.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2021-05-12T12:05:55.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:55.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2021-05-12T12:05:56.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:56.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2021-05-12T12:05:56.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:56.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2021-05-12T12:05:56.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:56.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2021-05-12T12:05:56.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:56.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2021-05-12T12:05:56.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:56.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2021-05-12T12:05:56.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:56.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2021-05-12T12:05:56.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:56.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2021-05-12T12:05:57.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:57.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2021-05-12T12:05:57.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:57.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2021-05-12T12:05:57.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:57.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2021-05-12T12:05:57.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:57.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2021-05-12T12:05:57.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:57.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2021-05-12T12:05:57.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:57.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2021-05-12T12:05:57.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:57.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2021-05-12T12:05:58.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:58.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2021-05-12T12:05:58.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:58.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2021-05-12T12:05:58.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:58.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2021-05-12T12:05:58.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:58.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2021-05-12T12:05:58.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:58.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2021-05-12T12:05:58.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:58.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2021-05-12T12:05:58.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:58.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2021-05-12T12:05:59.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:59.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2021-05-12T12:05:59.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:59.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2021-05-12T12:05:59.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:59.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2021-05-12T12:05:59.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:59.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2021-05-12T12:05:59.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:59.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2021-05-12T12:05:59.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:59.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2021-05-12T12:05:59.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:05:59.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2021-05-12T12:06:00.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:00.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2021-05-12T12:06:00.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:00.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2021-05-12T12:06:00.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:00.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2021-05-12T12:06:00.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:00.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2021-05-12T12:06:00.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:00.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2021-05-12T12:06:00.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:00.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2021-05-12T12:06:00.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:00.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2021-05-12T12:06:01.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:01.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2021-05-12T12:06:01.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:01.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2021-05-12T12:06:01.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:01.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2021-05-12T12:06:01.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:01.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2021-05-12T12:06:01.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:01.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2021-05-12T12:06:01.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:01.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2021-05-12T12:06:01.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:01.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2021-05-12T12:06:02.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:02.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2021-05-12T12:06:02.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:02.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2021-05-12T12:06:02.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:02.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2021-05-12T12:06:02.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:02.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2021-05-12T12:06:02.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:02.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2021-05-12T12:06:02.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:02.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2021-05-12T12:06:02.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:02.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2021-05-12T12:06:03.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:03.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2021-05-12T12:06:03.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:03.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2021-05-12T12:06:03.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:03.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2021-05-12T12:06:03.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:03.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2021-05-12T12:06:03.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:03.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2021-05-12T12:06:03.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:03.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2021-05-12T12:06:03.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:03.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2021-05-12T12:06:04.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:04.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2021-05-12T12:06:04.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:04.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2021-05-12T12:06:04.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:04.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2021-05-12T12:06:04.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:04.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2021-05-12T12:06:04.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:04.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2021-05-12T12:06:04.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:04.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2021-05-12T12:06:05.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:05.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2021-05-12T12:06:05.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:05.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2021-05-12T12:06:05.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:05.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2021-05-12T12:06:05.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:05.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2021-05-12T12:06:05.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:05.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2021-05-12T12:06:05.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:05.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2021-05-12T12:06:05.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:05.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2021-05-12T12:06:05.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:05.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2021-05-12T12:06:06.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:06.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2021-05-12T12:06:06.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:06.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2021-05-12T12:06:06.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:06.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2021-05-12T12:06:06.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:06.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2021-05-12T12:06:06.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:06.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2021-05-12T12:06:06.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:06.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2021-05-12T12:06:06.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:06.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2021-05-12T12:06:07.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:07.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2021-05-12T12:06:07.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:07.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2021-05-12T12:06:07.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:06:07.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2021-05-12T12:06:07.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:06:07.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:06:07.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:06:07.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:06:07.433 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:06:07.434 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:06:07.434 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:06:07.434 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:06:07.434 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:06:07.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:06:07.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:06:07.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:06:07.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:06:07.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:06:07.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:06:07.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:06:07.850 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:06:07.850 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:06:07.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:06:07.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:07.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:06:07.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836496 2021-05-12T12:06:07.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836496 2021-05-12T12:06:07.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496' 2021-05-12T12:06:07.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:07.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:06:08.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672974 2021-05-12T12:06:08.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672974 2021-05-12T12:06:08.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-05-12T12:06:08.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:08.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:06:08.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509452 2021-05-12T12:06:08.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509452 2021-05-12T12:06:08.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509452' 2021-05-12T12:06:08.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:08.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-05-12T12:06:08.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:08.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:06:08.261 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836496 2021-05-12T12:06:08.261 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:08.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836496 2021-05-12T12:06:08.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-05-12T12:06:08.262 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836496 2021-05-12T12:06:08.263 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:06:08.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836496 -lt 21474836496 2021-05-12T12:06:08.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:08.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-05-12T12:06:08.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:08.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:06:08.626 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672974 2021-05-12T12:06:08.626 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:08.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672974 2021-05-12T12:06:08.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-05-12T12:06:08.627 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672974 2021-05-12T12:06:08.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:06:09.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672974 -lt 42949672974 2021-05-12T12:06:09.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:09.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-05-12T12:06:09.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:09.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:06:09.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509452 2021-05-12T12:06:09.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:09.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509452 2021-05-12T12:06:09.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-05-12T12:06:09.011 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 64424509452 2021-05-12T12:06:09.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:06:09.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509450 -lt 64424509452 2021-05-12T12:06:09.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:06:10.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:06:10.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:06:10.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509452 -lt 64424509452 2021-05-12T12:06:10.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:06:10.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:10.777 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:11.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:06:11.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:11.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:11.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:11.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:11.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:11.261 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:11.261 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:11.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:06:11.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:11.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:11.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:12.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:06:12.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:06:12.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:06:12.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2021-05-12T12:06:12.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T12:06:12.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T12:06:12.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T12:06:12.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T12:06:12.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2021-05-12T12:06:12.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2021-05-12T12:06:12.469 INFO:tasks.workunit.client.0.smithi191.stdout:blackholing osds 0 2 2021-05-12T12:06:12.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:06:12.811 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:06:12.812 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T12:05:45.077119+0000 31'100 31:113 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:12.812 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:06:12.812 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:06:12.813 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:06:12.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:12.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2021-05-12T12:06:12.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:06:12.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:06:12.824 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:06:12.824 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:06:12.825 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:06:12.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.0.asok 2021-05-12T12:06:12.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:06:12.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T12:06:12.915 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:06:12.915 INFO:tasks.workunit.client.0.smithi191.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:06:12.915 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:06:12.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:12.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2021-05-12T12:06:12.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:06:12.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:06:12.925 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:06:12.925 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:06:12.925 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:06:12.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.2.asok 2021-05-12T12:06:12.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:06:12.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.2.asok config set objectstore_blackhole 1 2021-05-12T12:06:13.015 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:06:13.016 INFO:tasks.workunit.client.0.smithi191.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:06:13.016 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:06:13.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2021-05-12T12:06:13.024 INFO:tasks.workunit.client.0.smithi191.stdout:create new divergent objects 2021-05-12T12:06:13.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2021-05-12T12:06:13.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:06:13.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:06:13.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2021-05-12T12:06:13.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2021-05-12T12:06:13.026 INFO:tasks.workunit.client.0.smithi191.stdout:writing divergent objects 2021-05-12T12:06:13.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2021-05-12T12:06:13.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2021-05-12T12:06:13.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T12:06:13.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T12:06:13.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2021-05-12T12:06:13.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2021-05-12T12:06:13.029 INFO:tasks.workunit.client.0.smithi191.stdout:remove divergent objects 2021-05-12T12:06:13.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2021-05-12T12:06:13.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2021-05-12T12:06:13.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T12:06:13.031 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2021-05-12T12:06:13.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2021-05-12T12:06:13.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T12:06:13.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2021-05-12T12:06:13.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2021-05-12T12:06:13.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2021-05-12T12:06:13.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2021-05-12T12:06:13.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2021-05-12T12:06:23.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2021-05-12T12:06:23.034 INFO:tasks.workunit.client.0.smithi191.stderr:rados: no process found 2021-05-12T12:06:23.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2021-05-12T12:06:23.035 INFO:tasks.workunit.client.0.smithi191.stdout:killing all the osds 2021-05-12T12:06:23.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:06:23.394 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:06:23.394 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.861784+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.395 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862576+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.395 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.861550+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.395 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862816+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.395 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.861990+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.395 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862851+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.861655+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863357+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862150+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863136+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862175+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863387+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.861883+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868856+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862350+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868998+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.398 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.861989+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.398 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869326+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.398 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862383+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.398 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869666+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862422+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869661+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862169+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868397+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862674+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.400 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868646+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.400 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862368+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.400 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868930+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.400 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862613+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.400 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868840+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862619+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869001+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862411+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867093+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.862898+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.402 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867346+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.402 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863173+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.402 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867666+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.402 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863034+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867520+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863066+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867674+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863648+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.865063+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863695+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.865317+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863767+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.866699+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869044+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.405 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869472+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.405 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.868515+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.405 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.863874+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.405 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.868380+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864120+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.868520+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864615+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.865501+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864160+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.407 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.866894+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.407 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.863966+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.407 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.867200+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.407 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.863214+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.408 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.867082+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.408 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.863282+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.408 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.866992+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.408 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.863850+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.408 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864644+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864631+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864890+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864168+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864410+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864097+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.865064+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864503+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.450871+0000 31'106 33:124 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864918+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.865130+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.411 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.863860+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.411 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.865281+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.411 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.863663+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.411 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864917+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.865346+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864708+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.865502+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864451+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.865145+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.413 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 0 0 0 365 0 0 16 16 peering 2021-05-12T12:06:16.864510+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.413 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867837+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.413 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.864846+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.413 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867573+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.413 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.864064+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869378+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863637+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869272+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.863406+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.866962+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.415 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.866699+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.415 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.865523+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.415 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.866851+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.415 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.865282+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.415 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.865342+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.416 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867871+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.416 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.865109+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.416 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867737+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.416 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.864831+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.416 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867868+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.417 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867355+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.417 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867617+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.417 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867303+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.417 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867305+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.417 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867430+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.418 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869197+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.418 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867122+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.418 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869060+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.418 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.866864+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.419 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869126+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.419 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868793+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.419 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868831+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.419 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868624+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.419 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868629+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.420 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868728+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.420 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.870209+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.420 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.868437+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.420 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.870307+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.420 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.867823+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.421 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869510+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.421 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869557+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.421 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 0 0 0 364 0 0 16 16 peering 2021-05-12T12:06:16.869234+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:23.421 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:06:23.421 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:06:23.422 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:06:23.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2021-05-12T12:06:23.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:06:23.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:06:23.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:06:23.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:06:23.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:06:23.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:06:23.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-05-12T12:06:23.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2021-05-12T12:06:24.062 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already down. 2021-05-12T12:06:24.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-05-12T12:06:24.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2021-05-12T12:06:24.701 INFO:tasks.workunit.client.0.smithi191.stderr:osd.1 is already down. 2021-05-12T12:06:24.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-05-12T12:06:24.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2021-05-12T12:06:25.238 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already down. 2021-05-12T12:06:25.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:25.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2021-05-12T12:06:25.788 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already out. 2021-05-12T12:06:25.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:25.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2021-05-12T12:06:26.319 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already out. 2021-05-12T12:06:26.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2021-05-12T12:06:26.329 INFO:tasks.workunit.client.0.smithi191.stdout:bringing up non_divergent 0 2 2021-05-12T12:06:26.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:06:26.686 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:06:26.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.861784+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862576+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.861550+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862816+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.861990+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862851+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.861655+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.688 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863357+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.688 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862150+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.688 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863136+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.688 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862175+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.688 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863387+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.689 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.861883+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.689 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868856+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.689 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862350+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.689 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868998+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.690 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.861989+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.690 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869326+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.690 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862383+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.690 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869666+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.690 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862422+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.691 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869661+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.691 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862169+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.691 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868397+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.691 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862674+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.691 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868646+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.692 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862368+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.692 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868930+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.692 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862613+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.692 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868840+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.692 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862619+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.693 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869001+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.693 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862411+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.693 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867093+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.693 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.862898+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.694 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867346+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.694 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863173+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.694 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867666+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.694 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863034+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.694 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867520+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.695 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863066+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.695 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867674+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.695 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863648+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.695 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.865063+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.695 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863695+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.696 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.865317+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.696 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863767+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.696 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.866699+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.696 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869044+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.696 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869472+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.697 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.868515+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.697 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.863874+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.697 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.868380+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.697 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864120+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.697 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.868520+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.698 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864615+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.698 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.865501+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.698 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864160+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.698 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.866894+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.699 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.863966+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.699 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.867200+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.699 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.863214+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.699 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.867082+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.699 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.863282+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.700 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.866992+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.700 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.863850+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.700 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864644+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.700 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864631+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.700 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864890+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.701 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864168+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.701 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864410+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.701 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864097+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.701 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.865064+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.702 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864503+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.702 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.450871+0000 31'106 33:124 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.702 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864918+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.702 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.865130+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.702 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.863860+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.703 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.865281+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.703 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.863663+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.703 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864917+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.703 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.865346+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.703 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864708+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.704 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.865502+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.704 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864451+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.704 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.865145+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.704 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 0 0 0 365 0 0 16 16 stale+peering 2021-05-12T12:06:16.864510+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.704 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867837+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.705 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.864846+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.705 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867573+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.705 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.864064+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.705 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869378+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.705 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863637+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.706 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869272+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.706 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.863406+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.706 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.866962+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.706 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.866699+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.706 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.865523+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.707 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.866851+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.707 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.865282+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.707 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.865342+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.707 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867871+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.707 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.865109+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.708 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867737+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.708 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.864831+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.708 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867868+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.708 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867355+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.708 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867617+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.709 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867303+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.709 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867305+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.709 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867430+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.709 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869197+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.709 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867122+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.710 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869060+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.710 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.866864+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.710 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869126+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.710 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868793+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.710 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868831+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.711 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868624+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.711 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868629+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.711 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868728+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.711 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.870209+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.711 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.868437+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.712 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.870307+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.712 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.867823+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.712 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869510+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.712 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869557+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.712 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 0 0 0 364 0 0 16 16 stale+peering 2021-05-12T12:06:16.869234+0000 31'106 33:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:06:26.713 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:06:26.713 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:06:26.716 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:06:26.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:26.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2021-05-12T12:06:26.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:06:26.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:06:26.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:06:26.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:06:26.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:06:26.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:06:26.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:06:26.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:06:26.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:06:26.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:06:26.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:06:26.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:06:26.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:06:26.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:06:26.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:06:26.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:06:26.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:06:26.721 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:06:26.721 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:06:26.721 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:06:26.721 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:06:26.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:06:26.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:06:26.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:06:26.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:06:26.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:06:26.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:06:26.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:06:26.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:06:26.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:06:26.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:06:26.724 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:06:26.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:06:26.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:06:26.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:06:26.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:06:26.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:06:26.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:06:26.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:06:26.728 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:26.725+0000 7fb054a560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:26.730 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:26.727+0000 7fb054a560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:26.731 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:26.728+0000 7fb054a560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:27.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:06:27.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:06:27.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:06:27.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:06:27.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:06:27.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:27.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:06:27.067 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:06:27.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:27.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:06:27.249 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:27.246+0000 7fb054a560c0 -1 Falling back to public interface 2021-05-12T12:06:27.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:28.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:28.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:28.485 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:06:28.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:06:28.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:28.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:06:28.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:29.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:29.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:29.861 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:06:29.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:06:29.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:29.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:06:29.973 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:29.970+0000 7fb054a560c0 -1 osd.0 31 log_to_monitors {default=true} 2021-05-12T12:06:30.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:31.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:31.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:31.256 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:06:31.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:06:31.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:31.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:06:31.628 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up out weight 0 up_from 41 up_thru 0 down_at 35 last_clean_interval [5,31) [v2:127.0.0.1:6802/67634,v1:127.0.0.1:6803/67634] [v2:127.0.0.1:6804/67634,v1:127.0.0.1:6805/67634] exists,up efb617d0-a38f-4017-94d3-fc68ac2ed9be 2021-05-12T12:06:31.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:06:31.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:06:31.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:06:31.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:31.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2021-05-12T12:06:31.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:06:31.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:06:31.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:06:31.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:06:31.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:06:31.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:06:31.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:06:31.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:06:31.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:06:31.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:06:31.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:06:31.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:06:31.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:06:31.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:06:31.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:06:31.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:06:31.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:06:31.637 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:06:31.638 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:06:31.638 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:06:31.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:06:31.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:06:31.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:06:31.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:06:31.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:06:31.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:06:31.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:06:31.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:06:31.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:06:31.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:06:31.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:06:31.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:06:31.641 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:06:31.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:06:31.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:06:31.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:06:31.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:06:31.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:06:31.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:06:31.659 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:31.656+0000 7f6b7bd750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:31.660 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:31.658+0000 7f6b7bd750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:31.662 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:31.659+0000 7f6b7bd750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:32.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:06:32.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:06:32.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:06:32.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:06:32.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:06:32.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:32.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:06:32.003 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:06:32.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:32.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:32.178 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:32.176+0000 7f6b7bd750c0 -1 Falling back to public interface 2021-05-12T12:06:32.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:33.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:33.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:33.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:06:33.370 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:06:33.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:33.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:33.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:34.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:34.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:34.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:06:34.751 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:06:34.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:34.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:34.991 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:06:34.988+0000 7f6b7bd750c0 -1 osd.2 31 log_to_monitors {default=true} 2021-05-12T12:06:35.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:36.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:36.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:36.206 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:06:36.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:06:36.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:36.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:36.606 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up out weight 0 up_from 43 up_thru 0 down_at 37 last_clean_interval [15,31) [v2:127.0.0.1:6810/68217,v1:127.0.0.1:6811/68217] [v2:127.0.0.1:6812/68217,v1:127.0.0.1:6813/68217] exists,up 16b8b89a-4aa1-4296-88f5-d60bf3eabab1 2021-05-12T12:06:36.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:06:36.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:06:36.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:06:36.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:36.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2021-05-12T12:06:37.432 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already in. 2021-05-12T12:06:37.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:37.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2021-05-12T12:06:38.057 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already in. 2021-05-12T12:06:38.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:06:38.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2021-05-12T12:06:38.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:06:38.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:06:38.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:06:38.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:06:38.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:06:38.069 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:06:38.069 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:06:38.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:06:38.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:06:38.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:06:38.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:06:38.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:06:38.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:06:38.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:06:38.123 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:06:38.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:06:38.629 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:06:38.629 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:06:38.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:06:38.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:38.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:06:38.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=176093659140 2021-05-12T12:06:38.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 176093659140 2021-05-12T12:06:38.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-176093659140' 2021-05-12T12:06:38.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:38.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:06:38.872 INFO:tasks.workunit.client.0.smithi191.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:06:38.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:06:38.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:06:38.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:06:38.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:38.875 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:06:39.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=184683593731 2021-05-12T12:06:39.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 184683593731 2021-05-12T12:06:39.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-176093659140 2-184683593731' 2021-05-12T12:06:39.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:39.014 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-176093659140 2021-05-12T12:06:39.014 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:39.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:06:39.016 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-176093659140 2021-05-12T12:06:39.016 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:39.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=176093659140 2021-05-12T12:06:39.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2021-05-12T12:06:39.017 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 176093659140 2021-05-12T12:06:39.017 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:06:39.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659138 -lt 176093659140 2021-05-12T12:06:39.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:06:40.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:06:40.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:06:40.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659141 -lt 176093659140 2021-05-12T12:06:40.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:40.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-184683593731 2021-05-12T12:06:40.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:40.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:06:40.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-184683593731 2021-05-12T12:06:40.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:40.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=184683593731 2021-05-12T12:06:40.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 184683593731' 2021-05-12T12:06:40.792 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 184683593731 2021-05-12T12:06:40.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:06:41.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593732 -lt 184683593731 2021-05-12T12:06:41.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:06:41.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:41.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:41.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 128 == 0 2021-05-12T12:06:41.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:41.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:41.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:41.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:41.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:41.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:41.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:42.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:06:42.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:42.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:42.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:42.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:06:42.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:06:42.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:06:42.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:06:42.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:06:42.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:06:42.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:06:42.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:42.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:42.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:42.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:42.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:42.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:42.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:43.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:06:43.054 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:43.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:43.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:43.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:06:43.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:06:43.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:06:43.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:06:43.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:06:43.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:06:43.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:06:43.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:06:43.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:06:44.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:06:44.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:06:44.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:06:44.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:06:44.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:06:44.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:06:44.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:44.222 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:44.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:44.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:44.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:44.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:44.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:44.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:06:44.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:44.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:44.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:45.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:06:45.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:06:45.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:06:45.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:06:45.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:06:45.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:06:45.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:06:45.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:06:45.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:06:45.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:06:45.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:06:45.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:06:45.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:06:45.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:06:45.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:06:45.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:45.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:45.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:45.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:45.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:45.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:45.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:46.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:06:46.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:46.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:46.362 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:46.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:06:46.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:06:46.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:06:46.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:06:46.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:06:46.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:06:46.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:06:46.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:06:46.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:06:47.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:06:47.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:06:47.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:06:47.434 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:06:47.434 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:06:48.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:06:48.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:48.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:48.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:48.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:48.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:48.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:48.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:48.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:06:48.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:48.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:48.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:49.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:06:49.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:06:49.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:06:49.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:06:49.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:06:49.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:06:49.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:06:49.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:06:49.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:06:49.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:06:49.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:06:49.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:06:49.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:06:49.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:06:51.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:06:51.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:51.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:51.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:51.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:51.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:51.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:51.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:51.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:06:51.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:51.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:51.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:52.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:06:52.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:06:52.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:06:52.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:06:52.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:06:52.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:06:52.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:06:52.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:06:52.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:06:52.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:06:52.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:06:52.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:06:52.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:06:52.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:06:55.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:06:55.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:55.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:55.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:55.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:55.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:55.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:55.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:56.141 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:06:56.141 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:56.141 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:56.142 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:56.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:06:56.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:06:56.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:06:56.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:06:56.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:06:56.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:06:56.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:06:56.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:06:56.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:06:57.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:06:57.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:06:57.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:06:57.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:06:57.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:07:03.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:03.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:03.512 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:03.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:03.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:03.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:03.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:03.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:03.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:03.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:03.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:03.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:04.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:04.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:04.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:04.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:04.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:04.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:04.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:04.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:04.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:04.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:04.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:04.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:07:04.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:04.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:07:12.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:12.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:12.168 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:12.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:12.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:12.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:12.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:12.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:12.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:12.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:12.557 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:12.557 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:13.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:13.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:13.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:13.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:13.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:13.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:13.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:13.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:13.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:13.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:13.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:13.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:07:13.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:07:14.024 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:07:14.024 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_fingerprint": "43ec61a6-6986-4458-bee9-93f5dd8a4db2", 2021-05-12T12:07:14.025 INFO:tasks.workunit.client.0.smithi191.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:14.025 INFO:tasks.workunit.client.0.smithi191.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:07:14.025 INFO:tasks.workunit.client.0.smithi191.stdout: "timestamp": "2021-05-12T12:07:14.021151+0000", 2021-05-12T12:07:14.025 INFO:tasks.workunit.client.0.smithi191.stdout: "tag": "", 2021-05-12T12:07:14.026 INFO:tasks.workunit.client.0.smithi191.stdout: "health": { 2021-05-12T12:07:14.026 INFO:tasks.workunit.client.0.smithi191.stdout: "status": "HEALTH_ERR", 2021-05-12T12:07:14.026 INFO:tasks.workunit.client.0.smithi191.stdout: "checks": { 2021-05-12T12:07:14.026 INFO:tasks.workunit.client.0.smithi191.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:07:14.026 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:07:14.027 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:14.027 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:07:14.027 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:07:14.027 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.028 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:07:14.028 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.028 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:07:14.028 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.028 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.029 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:14.029 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.029 INFO:tasks.workunit.client.0.smithi191.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:07:14.029 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:14.030 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:14.030 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:07:14.030 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 17 2021-05-12T12:07:14.030 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.030 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [], 2021-05-12T12:07:14.031 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:14.031 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.031 INFO:tasks.workunit.client.0.smithi191.stdout: "OSD_DOWN": { 2021-05-12T12:07:14.031 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:14.032 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:14.032 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 osds down", 2021-05-12T12:07:14.032 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:07:14.032 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.032 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:07:14.033 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.033 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "osd.1 (root=default,host=smithi191) is down" 2021-05-12T12:07:14.033 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.033 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.033 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:14.034 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.034 INFO:tasks.workunit.client.0.smithi191.stdout: "PG_DEGRADED": { 2021-05-12T12:07:14.034 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:14.034 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:14.035 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 100 pgs degraded", 2021-05-12T12:07:14.035 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 100 2021-05-12T12:07:14.035 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.035 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:07:14.035 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.036 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.036 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.036 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.036 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.037 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.037 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.037 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.037 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.037 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.038 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.038 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.038 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.039 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.039 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.039 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.039 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.039 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.040 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.040 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.040 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.040 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.041 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.041 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.041 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.041 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.041 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.042 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.042 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.042 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.042 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.043 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.043 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.043 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.043 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.044 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.044 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.044 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.044 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.045 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.045 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.045 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.045 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.046 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.046 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.046 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.046 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.046 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.047 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.047 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.047 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.047 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.047 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.048 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.048 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.048 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.048 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.048 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.049 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.049 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.049 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.050 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.050 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.050 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.050 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.050 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.051 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.051 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.051 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.051 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.052 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.052 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.052 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.052 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.20 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.052 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.053 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.053 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.21 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.053 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.054 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.054 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.22 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.054 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.054 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.055 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.23 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.055 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.055 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.055 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.4e is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.056 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.056 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.056 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.4f is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.056 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.056 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.057 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.50 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.057 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.057 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.057 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.51 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.058 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.058 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.058 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.52 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.058 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.058 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.059 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.53 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.059 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.059 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.059 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.54 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.060 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.060 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.060 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.55 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.060 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.060 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.061 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.56 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.061 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.061 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.061 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.57 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.061 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.062 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.062 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.58 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.062 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.062 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.063 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.59 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.063 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.063 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.063 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5a is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.064 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.064 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.064 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5b is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.064 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.064 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.065 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5c is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.065 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.065 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.065 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5d is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.065 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.066 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.066 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5e is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.066 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.066 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.067 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5f is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.067 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.067 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.067 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.60 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.067 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.068 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.068 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.61 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.068 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.068 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.068 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.62 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.069 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.069 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.069 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.63 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.069 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.070 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.070 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.64 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:14.070 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.070 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.070 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:14.071 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.071 INFO:tasks.workunit.client.0.smithi191.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:07:14.071 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:14.071 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:14.072 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:07:14.072 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:07:14.072 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.072 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:07:14.072 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.073 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:07:14.073 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.073 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.073 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:07:14.074 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.074 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.074 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:14.074 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.074 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.075 INFO:tasks.workunit.client.0.smithi191.stdout: "mutes": [] 2021-05-12T12:07:14.075 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.075 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_first_committed": 1, 2021-05-12T12:07:14.075 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_last_committed": 1, 2021-05-12T12:07:14.076 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap": { 2021-05-12T12:07:14.076 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:07:14.076 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:07:14.076 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:05:07.934785Z", 2021-05-12T12:07:14.076 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:05:07.934785Z", 2021-05-12T12:07:14.077 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release": 17, 2021-05-12T12:07:14.077 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:07:14.077 INFO:tasks.workunit.client.0.smithi191.stdout: "election_strategy": 1, 2021-05-12T12:07:14.077 INFO:tasks.workunit.client.0.smithi191.stdout: "disallowed_leaders: ": "", 2021-05-12T12:07:14.077 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": false, 2021-05-12T12:07:14.078 INFO:tasks.workunit.client.0.smithi191.stdout: "features": { 2021-05-12T12:07:14.078 INFO:tasks.workunit.client.0.smithi191.stdout: "persistent": [ 2021-05-12T12:07:14.078 INFO:tasks.workunit.client.0.smithi191.stdout: "kraken", 2021-05-12T12:07:14.078 INFO:tasks.workunit.client.0.smithi191.stdout: "luminous", 2021-05-12T12:07:14.079 INFO:tasks.workunit.client.0.smithi191.stdout: "mimic", 2021-05-12T12:07:14.079 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap-prune", 2021-05-12T12:07:14.079 INFO:tasks.workunit.client.0.smithi191.stdout: "nautilus", 2021-05-12T12:07:14.079 INFO:tasks.workunit.client.0.smithi191.stdout: "octopus", 2021-05-12T12:07:14.079 INFO:tasks.workunit.client.0.smithi191.stdout: "pacific", 2021-05-12T12:07:14.080 INFO:tasks.workunit.client.0.smithi191.stdout: "elector-pinging", 2021-05-12T12:07:14.080 INFO:tasks.workunit.client.0.smithi191.stdout: "quincy" 2021-05-12T12:07:14.080 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.080 INFO:tasks.workunit.client.0.smithi191.stdout: "optional": [] 2021-05-12T12:07:14.081 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.081 INFO:tasks.workunit.client.0.smithi191.stdout: "mons": [ 2021-05-12T12:07:14.081 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.081 INFO:tasks.workunit.client.0.smithi191.stdout: "rank": 0, 2021-05-12T12:07:14.081 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "a", 2021-05-12T12:07:14.082 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:07:14.082 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.082 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.082 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.082 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:07:14.083 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 0 2021-05-12T12:07:14.083 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.083 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.083 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.084 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:07:14.084 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:07:14.084 INFO:tasks.workunit.client.0.smithi191.stdout: "priority": 0, 2021-05-12T12:07:14.084 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 0, 2021-05-12T12:07:14.084 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_location": "{}" 2021-05-12T12:07:14.085 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.085 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.085 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.085 INFO:tasks.workunit.client.0.smithi191.stdout: "quorum": [ 2021-05-12T12:07:14.085 INFO:tasks.workunit.client.0.smithi191.stdout: 0 2021-05-12T12:07:14.086 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.086 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap": { 2021-05-12T12:07:14.086 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 48, 2021-05-12T12:07:14.086 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:07:14.087 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:05:08.774093+0000", 2021-05-12T12:07:14.087 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:06:38.214829+0000", 2021-05-12T12:07:14.087 INFO:tasks.workunit.client.0.smithi191.stdout: "last_up_change": "2021-05-12T12:06:35.204550+0000", 2021-05-12T12:07:14.087 INFO:tasks.workunit.client.0.smithi191.stdout: "last_in_change": "2021-05-12T12:06:37.743635+0000", 2021-05-12T12:07:14.087 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:07:14.088 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_num": 5800384, 2021-05-12T12:07:14.088 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_set": [ 2021-05-12T12:07:14.088 INFO:tasks.workunit.client.0.smithi191.stdout: "nodown", 2021-05-12T12:07:14.088 INFO:tasks.workunit.client.0.smithi191.stdout: "noin", 2021-05-12T12:07:14.088 INFO:tasks.workunit.client.0.smithi191.stdout: "noout", 2021-05-12T12:07:14.089 INFO:tasks.workunit.client.0.smithi191.stdout: "pglog_hardlimit", 2021-05-12T12:07:14.089 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snapdirs", 2021-05-12T12:07:14.089 INFO:tasks.workunit.client.0.smithi191.stdout: "recovery_deletes", 2021-05-12T12:07:14.089 INFO:tasks.workunit.client.0.smithi191.stdout: "sortbitwise" 2021-05-12T12:07:14.089 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.090 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_version": 9, 2021-05-12T12:07:14.090 INFO:tasks.workunit.client.0.smithi191.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:07:14.090 INFO:tasks.workunit.client.0.smithi191.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:07:14.090 INFO:tasks.workunit.client.0.smithi191.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:07:14.091 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_snapshot": "", 2021-05-12T12:07:14.091 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_max": 1, 2021-05-12T12:07:14.091 INFO:tasks.workunit.client.0.smithi191.stdout: "max_osd": 3, 2021-05-12T12:07:14.091 INFO:tasks.workunit.client.0.smithi191.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:07:14.091 INFO:tasks.workunit.client.0.smithi191.stdout: "min_compat_client": "jewel", 2021-05-12T12:07:14.092 INFO:tasks.workunit.client.0.smithi191.stdout: "require_osd_release": "quincy", 2021-05-12T12:07:14.093 INFO:tasks.workunit.client.0.smithi191.stdout: "pools": [ 2021-05-12T12:07:14.094 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.094 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:07:14.094 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_name": "test", 2021-05-12T12:07:14.094 INFO:tasks.workunit.client.0.smithi191.stdout: "create_time": "2021-05-12T12:05:43.641277+0000", 2021-05-12T12:07:14.095 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": 1, 2021-05-12T12:07:14.095 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_names": "hashpspool", 2021-05-12T12:07:14.095 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:07:14.095 INFO:tasks.workunit.client.0.smithi191.stdout: "size": 3, 2021-05-12T12:07:14.096 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 2, 2021-05-12T12:07:14.096 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_rule": 0, 2021-05-12T12:07:14.096 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:07:14.096 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:07:14.096 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:07:14.097 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:07:14.097 INFO:tasks.workunit.client.0.smithi191.stdout: "object_hash": 2, 2021-05-12T12:07:14.097 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_autoscale_mode": "on", 2021-05-12T12:07:14.097 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num": 128, 2021-05-12T12:07:14.098 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num": 1, 2021-05-12T12:07:14.098 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num_target": 128, 2021-05-12T12:07:14.098 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_target": 128, 2021-05-12T12:07:14.098 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_pending": 128, 2021-05-12T12:07:14.098 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pg_merge_meta": { 2021-05-12T12:07:14.099 INFO:tasks.workunit.client.0.smithi191.stdout: "source_pgid": "0.0", 2021-05-12T12:07:14.099 INFO:tasks.workunit.client.0.smithi191.stdout: "ready_epoch": 0, 2021-05-12T12:07:14.099 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_started": 0, 2021-05-12T12:07:14.099 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": 0, 2021-05-12T12:07:14.100 INFO:tasks.workunit.client.0.smithi191.stdout: "source_version": "0'0", 2021-05-12T12:07:14.100 INFO:tasks.workunit.client.0.smithi191.stdout: "target_version": "0'0" 2021-05-12T12:07:14.100 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.100 INFO:tasks.workunit.client.0.smithi191.stdout: "last_change": "33", 2021-05-12T12:07:14.100 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend": "0", 2021-05-12T12:07:14.101 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:07:14.101 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_preluminous": "33", 2021-05-12T12:07:14.101 INFO:tasks.workunit.client.0.smithi191.stdout: "auid": 0, 2021-05-12T12:07:14.101 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:07:14.101 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_seq": 0, 2021-05-12T12:07:14.102 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_epoch": 0, 2021-05-12T12:07:14.102 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_snaps": [], 2021-05-12T12:07:14.102 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps": "[]", 2021-05-12T12:07:14.102 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_bytes": 0, 2021-05-12T12:07:14.103 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_objects": 0, 2021-05-12T12:07:14.103 INFO:tasks.workunit.client.0.smithi191.stdout: "tiers": [], 2021-05-12T12:07:14.103 INFO:tasks.workunit.client.0.smithi191.stdout: "tier_of": -1, 2021-05-12T12:07:14.103 INFO:tasks.workunit.client.0.smithi191.stdout: "read_tier": -1, 2021-05-12T12:07:14.103 INFO:tasks.workunit.client.0.smithi191.stdout: "write_tier": -1, 2021-05-12T12:07:14.104 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_mode": "none", 2021-05-12T12:07:14.104 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_bytes": 0, 2021-05-12T12:07:14.104 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_objects": 0, 2021-05-12T12:07:14.104 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:07:14.104 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:07:14.105 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:07:14.105 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_flush_age": 0, 2021-05-12T12:07:14.105 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_evict_age": 0, 2021-05-12T12:07:14.105 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profile": "", 2021-05-12T12:07:14.106 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_params": { 2021-05-12T12:07:14.106 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "none" 2021-05-12T12:07:14.106 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.106 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_period": 0, 2021-05-12T12:07:14.107 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_count": 0, 2021-05-12T12:07:14.107 INFO:tasks.workunit.client.0.smithi191.stdout: "use_gmt_hitset": true, 2021-05-12T12:07:14.107 INFO:tasks.workunit.client.0.smithi191.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:07:14.107 INFO:tasks.workunit.client.0.smithi191.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:07:14.107 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:07:14.108 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:07:14.108 INFO:tasks.workunit.client.0.smithi191.stdout: "grade_table": [], 2021-05-12T12:07:14.108 INFO:tasks.workunit.client.0.smithi191.stdout: "stripe_width": 0, 2021-05-12T12:07:14.108 INFO:tasks.workunit.client.0.smithi191.stdout: "expected_num_objects": 0, 2021-05-12T12:07:14.108 INFO:tasks.workunit.client.0.smithi191.stdout: "fast_read": false, 2021-05-12T12:07:14.109 INFO:tasks.workunit.client.0.smithi191.stdout: "options": {}, 2021-05-12T12:07:14.109 INFO:tasks.workunit.client.0.smithi191.stdout: "application_metadata": {} 2021-05-12T12:07:14.109 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.109 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.110 INFO:tasks.workunit.client.0.smithi191.stdout: "osds": [ 2021-05-12T12:07:14.110 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.110 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:07:14.110 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "efb617d0-a38f-4017-94d3-fc68ac2ed9be", 2021-05-12T12:07:14.110 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:07:14.111 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:07:14.111 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:07:14.111 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:07:14.111 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 5, 2021-05-12T12:07:14.111 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 31, 2021-05-12T12:07:14.112 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 41, 2021-05-12T12:07:14.112 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 47, 2021-05-12T12:07:14.112 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 35, 2021-05-12T12:07:14.112 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:07:14.113 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:07:14.113 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.113 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.113 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.113 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:07:14.114 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:14.114 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.114 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.114 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.115 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:07:14.115 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:14.115 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.115 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.116 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.116 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:07:14.116 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.116 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.116 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.117 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:07:14.117 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:14.117 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.117 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.118 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.118 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:07:14.118 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:14.118 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.118 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.119 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.119 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:07:14.119 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.119 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.120 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.120 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:07:14.120 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:14.120 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.120 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.121 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.121 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:07:14.121 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:14.121 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.122 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.122 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.122 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:07:14.122 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.122 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.123 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.123 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:07:14.123 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:14.123 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.124 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.124 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.124 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:07:14.124 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:14.124 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.125 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.125 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.125 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6803/67634", 2021-05-12T12:07:14.125 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6805/67634", 2021-05-12T12:07:14.126 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6809/67634", 2021-05-12T12:07:14.126 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6807/67634", 2021-05-12T12:07:14.126 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:07:14.126 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:07:14.126 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:07:14.127 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.127 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.127 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.127 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:07:14.127 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "5fe20a5a-c3fb-42c5-aadd-2c4671a13114", 2021-05-12T12:07:14.128 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 0, 2021-05-12T12:07:14.128 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:07:14.128 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:07:14.128 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:07:14.129 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 0, 2021-05-12T12:07:14.129 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 0, 2021-05-12T12:07:14.129 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 10, 2021-05-12T12:07:14.129 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 33, 2021-05-12T12:07:14.129 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 36, 2021-05-12T12:07:14.130 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:07:14.130 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:07:14.130 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.130 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.131 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.136 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:07:14.136 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:14.136 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.136 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.137 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.137 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:07:14.137 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:14.137 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.138 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.138 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.138 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:07:14.138 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.138 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.139 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.139 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:07:14.139 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:14.139 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.140 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.140 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.140 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:07:14.140 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:14.140 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.141 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.141 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.141 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:07:14.141 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.141 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.142 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.142 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:07:14.142 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:14.142 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.143 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.143 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.143 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:07:14.143 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:14.143 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.144 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.144 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.144 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:07:14.144 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.144 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.145 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.145 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:07:14.145 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:14.145 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.146 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.146 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.146 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:07:14.146 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:14.146 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.147 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.147 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.147 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/62131", 2021-05-12T12:07:14.147 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/62131", 2021-05-12T12:07:14.148 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/62131", 2021-05-12T12:07:14.148 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/62131", 2021-05-12T12:07:14.148 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:07:14.148 INFO:tasks.workunit.client.0.smithi191.stdout: "exists" 2021-05-12T12:07:14.148 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.149 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.149 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.149 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:07:14.149 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "16b8b89a-4aa1-4296-88f5-d60bf3eabab1", 2021-05-12T12:07:14.149 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:07:14.150 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:07:14.150 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:07:14.150 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:07:14.150 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 15, 2021-05-12T12:07:14.151 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 31, 2021-05-12T12:07:14.151 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 43, 2021-05-12T12:07:14.151 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 0, 2021-05-12T12:07:14.151 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 37, 2021-05-12T12:07:14.151 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:07:14.152 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:07:14.152 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.152 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.152 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.153 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:07:14.153 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:14.153 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.153 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.153 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.154 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:07:14.154 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:14.154 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.154 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.155 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.155 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:07:14.155 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.155 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.155 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.156 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:07:14.156 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:14.156 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.156 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.157 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.157 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:07:14.157 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:14.157 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.157 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.158 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.158 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:07:14.158 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.158 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.158 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.159 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:07:14.159 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:14.159 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.159 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.160 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.160 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:07:14.160 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:14.160 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.160 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.161 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.161 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:07:14.161 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:14.161 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.162 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:14.162 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:07:14.162 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:14.162 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.163 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.163 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:14.163 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:07:14.163 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:14.163 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.164 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.164 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.164 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/68217", 2021-05-12T12:07:14.164 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/68217", 2021-05-12T12:07:14.165 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/68217", 2021-05-12T12:07:14.165 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/68217", 2021-05-12T12:07:14.165 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:07:14.165 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:07:14.165 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:07:14.166 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.166 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.166 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.166 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_xinfo": [ 2021-05-12T12:07:14.167 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.167 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:07:14.167 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:06:23.845446+0000", 2021-05-12T12:07:14.167 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:07:14.167 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:07:14.168 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:07:14.168 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:07:14.168 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:19.111982+0000", 2021-05-12T12:07:14.168 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:07:14.169 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.169 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.169 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:07:14.169 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:06:24.454727+0000", 2021-05-12T12:07:14.169 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:07:14.170 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:07:14.170 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:07:14.170 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:07:14.170 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:28.849900+0000", 2021-05-12T12:07:14.171 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:07:14.171 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.171 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.171 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:07:14.171 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:06:25.022069+0000", 2021-05-12T12:07:14.172 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:07:14.172 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:07:14.172 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:07:14.172 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:07:14.172 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:40.000052+0000", 2021-05-12T12:07:14.173 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:07:14.173 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.173 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.173 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap": [], 2021-05-12T12:07:14.173 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap_items": [], 2021-05-12T12:07:14.174 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_temp": [], 2021-05-12T12:07:14.174 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_temp": [], 2021-05-12T12:07:14.174 INFO:tasks.workunit.client.0.smithi191.stdout: "blocklist": {}, 2021-05-12T12:07:14.174 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profiles": { 2021-05-12T12:07:14.175 INFO:tasks.workunit.client.0.smithi191.stdout: "default": { 2021-05-12T12:07:14.175 INFO:tasks.workunit.client.0.smithi191.stdout: "k": "2", 2021-05-12T12:07:14.175 INFO:tasks.workunit.client.0.smithi191.stdout: "m": "2", 2021-05-12T12:07:14.175 INFO:tasks.workunit.client.0.smithi191.stdout: "plugin": "jerasure", 2021-05-12T12:07:14.175 INFO:tasks.workunit.client.0.smithi191.stdout: "technique": "reed_sol_van" 2021-05-12T12:07:14.176 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.176 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.176 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps_queue": [], 2021-05-12T12:07:14.176 INFO:tasks.workunit.client.0.smithi191.stdout: "new_removed_snaps": [], 2021-05-12T12:07:14.176 INFO:tasks.workunit.client.0.smithi191.stdout: "new_purged_snaps": [], 2021-05-12T12:07:14.177 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_node_flags": {}, 2021-05-12T12:07:14.177 INFO:tasks.workunit.client.0.smithi191.stdout: "device_class_flags": {}, 2021-05-12T12:07:14.177 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": { 2021-05-12T12:07:14.177 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_enabled": false, 2021-05-12T12:07:14.177 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_bucket_count": 0, 2021-05-12T12:07:14.178 INFO:tasks.workunit.client.0.smithi191.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:07:14.178 INFO:tasks.workunit.client.0.smithi191.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:07:14.178 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_bucket": 0 2021-05-12T12:07:14.178 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.179 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.179 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_metadata": [ 2021-05-12T12:07:14.179 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.179 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:14.179 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:07:14.180 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6804/67634,v1:127.0.0.1:6805/67634]", 2021-05-12T12:07:14.180 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:07:14.180 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:07:14.180 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:07:14.181 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:07:14.181 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:07:14.181 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:07:14.181 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:07:14.181 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:07:14.182 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:07:14.182 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:07:14.182 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:07:14.182 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:07:14.182 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:07:14.183 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:07:14.183 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:07:14.183 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:07:14.183 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:14.184 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:07:14.184 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:07:14.184 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:07:14.184 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:07:14.184 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:07:14.185 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:07:14.185 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:07:14.185 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:07:14.185 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6802/67634,v1:127.0.0.1:6803/67634]", 2021-05-12T12:07:14.185 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:07:14.186 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/67634,v1:127.0.0.1:6809/67634]", 2021-05-12T12:07:14.186 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/67634,v1:127.0.0.1:6807/67634]", 2021-05-12T12:07:14.186 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:07:14.186 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:07:14.186 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:07:14.187 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:07:14.187 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:07:14.187 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:07:14.187 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:07:14.188 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:07:14.188 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:07:14.188 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:07:14.188 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:07:14.188 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:07:14.189 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:07:14.189 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.189 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.189 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:14.189 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:07:14.190 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/62131,v1:127.0.0.1:6813/62131]", 2021-05-12T12:07:14.190 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:07:14.190 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:07:14.190 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:07:14.191 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:07:14.191 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:07:14.191 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:07:14.191 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:07:14.191 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:07:14.192 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:07:14.192 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:07:14.192 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:07:14.192 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:07:14.193 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:07:14.193 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:07:14.193 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:07:14.193 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:07:14.193 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:14.194 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:07:14.194 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:07:14.194 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:07:14.194 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:07:14.195 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:07:14.195 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:07:14.195 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:07:14.195 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:07:14.195 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/62131,v1:127.0.0.1:6811/62131]", 2021-05-12T12:07:14.196 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:07:14.196 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/62131,v1:127.0.0.1:6817/62131]", 2021-05-12T12:07:14.196 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/62131,v1:127.0.0.1:6815/62131]", 2021-05-12T12:07:14.196 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:07:14.196 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:07:14.197 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:07:14.197 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:07:14.197 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:07:14.197 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:07:14.198 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:07:14.198 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:07:14.198 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:07:14.198 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:07:14.198 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:07:14.199 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:07:14.199 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:07:14.199 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.199 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.199 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:14.200 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:07:14.200 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/68217,v1:127.0.0.1:6813/68217]", 2021-05-12T12:07:14.200 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:07:14.200 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:07:14.200 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:07:14.201 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:07:14.201 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:07:14.201 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:07:14.201 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:07:14.201 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:07:14.202 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:07:14.202 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:07:14.202 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:07:14.202 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:07:14.203 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:07:14.203 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:07:14.203 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:07:14.203 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:07:14.203 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:14.204 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:07:14.204 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:07:14.204 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:07:14.204 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:07:14.204 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:07:14.205 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:07:14.205 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:07:14.205 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:07:14.205 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/68217,v1:127.0.0.1:6811/68217]", 2021-05-12T12:07:14.205 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:07:14.206 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/68217,v1:127.0.0.1:6817/68217]", 2021-05-12T12:07:14.206 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/68217,v1:127.0.0.1:6815/68217]", 2021-05-12T12:07:14.206 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:07:14.206 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:07:14.207 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:07:14.207 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:07:14.207 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:07:14.207 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:07:14.207 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:07:14.208 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:07:14.208 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:07:14.208 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:07:14.208 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:07:14.209 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:07:14.209 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:07:14.209 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.209 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.209 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_clean_epochs": { 2021-05-12T12:07:14.210 INFO:tasks.workunit.client.0.smithi191.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:07:14.210 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": { 2021-05-12T12:07:14.210 INFO:tasks.workunit.client.0.smithi191.stdout: "per_pool": [] 2021-05-12T12:07:14.210 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.210 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_epochs": [ 2021-05-12T12:07:14.211 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.211 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:14.211 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 41 2021-05-12T12:07:14.211 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.212 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.212 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:14.212 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 10 2021-05-12T12:07:14.212 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.212 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.213 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:14.213 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 43 2021-05-12T12:07:14.213 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.213 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.213 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.214 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_first_committed": 1, 2021-05-12T12:07:14.214 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_last_committed": 48, 2021-05-12T12:07:14.214 INFO:tasks.workunit.client.0.smithi191.stdout: "crushmap": { 2021-05-12T12:07:14.214 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": [ 2021-05-12T12:07:14.214 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.215 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:14.215 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.0", 2021-05-12T12:07:14.215 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:07:14.215 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.215 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.216 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:14.216 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.1", 2021-05-12T12:07:14.216 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:07:14.216 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.217 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.217 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:14.217 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.2", 2021-05-12T12:07:14.217 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:07:14.217 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.218 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.218 INFO:tasks.workunit.client.0.smithi191.stdout: "types": [ 2021-05-12T12:07:14.218 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.218 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 0, 2021-05-12T12:07:14.218 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd" 2021-05-12T12:07:14.219 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.219 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.219 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:07:14.219 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "host" 2021-05-12T12:07:14.219 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.220 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.220 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 2, 2021-05-12T12:07:14.220 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "chassis" 2021-05-12T12:07:14.220 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.221 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.221 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 3, 2021-05-12T12:07:14.221 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "rack" 2021-05-12T12:07:14.221 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.221 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.222 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 4, 2021-05-12T12:07:14.222 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "row" 2021-05-12T12:07:14.222 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.222 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.222 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 5, 2021-05-12T12:07:14.223 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pdu" 2021-05-12T12:07:14.223 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.223 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.223 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 6, 2021-05-12T12:07:14.223 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pod" 2021-05-12T12:07:14.224 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.224 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.224 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 7, 2021-05-12T12:07:14.224 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "room" 2021-05-12T12:07:14.225 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.225 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.225 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 8, 2021-05-12T12:07:14.225 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "datacenter" 2021-05-12T12:07:14.225 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.226 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.226 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 9, 2021-05-12T12:07:14.226 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "zone" 2021-05-12T12:07:14.226 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.226 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.227 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 10, 2021-05-12T12:07:14.227 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "region" 2021-05-12T12:07:14.227 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.227 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.227 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:07:14.228 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "root" 2021-05-12T12:07:14.228 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.228 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.228 INFO:tasks.workunit.client.0.smithi191.stdout: "buckets": [ 2021-05-12T12:07:14.229 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.229 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -1, 2021-05-12T12:07:14.229 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default", 2021-05-12T12:07:14.229 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:07:14.229 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:07:14.230 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:14.230 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:07:14.230 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:07:14.230 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:07:14.230 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.231 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:07:14.231 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:14.231 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:07:14.231 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.232 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.232 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.232 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.232 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -2, 2021-05-12T12:07:14.232 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default~hdd", 2021-05-12T12:07:14.233 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:07:14.233 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:07:14.233 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:14.233 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:07:14.233 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:07:14.234 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:07:14.234 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.234 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:07:14.234 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:14.234 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:07:14.235 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.235 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.235 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.235 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.235 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:07:14.236 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191", 2021-05-12T12:07:14.236 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:07:14.236 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:07:14.236 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:14.237 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:07:14.237 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:07:14.237 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:07:14.237 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.237 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:14.238 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:14.238 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:07:14.238 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.238 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.238 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:14.239 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:14.239 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:07:14.239 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.239 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.239 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:14.240 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:14.240 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:07:14.240 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.240 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.241 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.241 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.241 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:07:14.241 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191~hdd", 2021-05-12T12:07:14.241 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:07:14.242 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:07:14.242 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:14.242 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:07:14.242 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:07:14.242 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:07:14.243 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.243 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:14.243 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:14.243 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:07:14.243 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.244 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.244 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:14.244 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:14.244 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:07:14.244 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.245 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.245 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:14.245 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:14.245 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:07:14.246 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.246 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.246 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.246 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.246 INFO:tasks.workunit.client.0.smithi191.stdout: "rules": [ 2021-05-12T12:07:14.247 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.247 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 0, 2021-05-12T12:07:14.247 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "replicated_rule", 2021-05-12T12:07:14.247 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 0, 2021-05-12T12:07:14.247 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:07:14.248 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 1, 2021-05-12T12:07:14.248 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 10, 2021-05-12T12:07:14.248 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2021-05-12T12:07:14.248 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.248 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2021-05-12T12:07:14.249 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2021-05-12T12:07:14.249 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2021-05-12T12:07:14.249 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.249 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.249 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_firstn", 2021-05-12T12:07:14.250 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2021-05-12T12:07:14.250 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2021-05-12T12:07:14.250 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.250 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.251 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2021-05-12T12:07:14.251 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.251 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:14.251 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.251 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.252 INFO:tasks.workunit.client.0.smithi191.stdout: "tunables": { 2021-05-12T12:07:14.252 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_tries": 0, 2021-05-12T12:07:14.252 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:07:14.252 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_total_tries": 50, 2021-05-12T12:07:14.252 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:07:14.253 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:07:14.253 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_stable": 1, 2021-05-12T12:07:14.253 INFO:tasks.workunit.client.0.smithi191.stdout: "straw_calc_version": 1, 2021-05-12T12:07:14.253 INFO:tasks.workunit.client.0.smithi191.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:07:14.253 INFO:tasks.workunit.client.0.smithi191.stdout: "profile": "jewel", 2021-05-12T12:07:14.254 INFO:tasks.workunit.client.0.smithi191.stdout: "optimal_tunables": 1, 2021-05-12T12:07:14.254 INFO:tasks.workunit.client.0.smithi191.stdout: "legacy_tunables": 0, 2021-05-12T12:07:14.254 INFO:tasks.workunit.client.0.smithi191.stdout: "minimum_required_version": "jewel", 2021-05-12T12:07:14.254 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables": 1, 2021-05-12T12:07:14.255 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables2": 1, 2021-05-12T12:07:14.255 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v2_rules": 0, 2021-05-12T12:07:14.255 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables3": 1, 2021-05-12T12:07:14.255 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v3_rules": 0, 2021-05-12T12:07:14.255 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v4_buckets": 1, 2021-05-12T12:07:14.256 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables5": 1, 2021-05-12T12:07:14.256 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v5_rules": 0 2021-05-12T12:07:14.256 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.256 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_args": {} 2021-05-12T12:07:14.256 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.257 INFO:tasks.workunit.client.0.smithi191.stdout: "fsmap": { 2021-05-12T12:07:14.257 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:07:14.257 INFO:tasks.workunit.client.0.smithi191.stdout: "default_fscid": -1, 2021-05-12T12:07:14.257 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": { 2021-05-12T12:07:14.257 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": {}, 2021-05-12T12:07:14.258 INFO:tasks.workunit.client.0.smithi191.stdout: "ro_compat": {}, 2021-05-12T12:07:14.258 INFO:tasks.workunit.client.0.smithi191.stdout: "incompat": { 2021-05-12T12:07:14.258 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_1": "base v0.20", 2021-05-12T12:07:14.258 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:07:14.258 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:07:14.259 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:07:14.259 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:07:14.259 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:07:14.259 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_8": "no anchor table", 2021-05-12T12:07:14.260 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_9": "file layout v2", 2021-05-12T12:07:14.260 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:07:14.260 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.260 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.260 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_flags": { 2021-05-12T12:07:14.261 INFO:tasks.workunit.client.0.smithi191.stdout: "enable_multiple": true, 2021-05-12T12:07:14.261 INFO:tasks.workunit.client.0.smithi191.stdout: "ever_enabled_multiple": true 2021-05-12T12:07:14.261 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.261 INFO:tasks.workunit.client.0.smithi191.stdout: "standbys": [], 2021-05-12T12:07:14.261 INFO:tasks.workunit.client.0.smithi191.stdout: "filesystems": [] 2021-05-12T12:07:14.262 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.262 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:07:14.262 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:07:14.262 INFO:tasks.workunit.client.0.smithi191.stdout: "auth": { 2021-05-12T12:07:14.262 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:07:14.263 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 5, 2021-05-12T12:07:14.263 INFO:tasks.workunit.client.0.smithi191.stdout: "num_secrets": 3 2021-05-12T12:07:14.263 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.263 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 128, 2021-05-12T12:07:14.263 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_active": 128, 2021-05-12T12:07:14.264 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_unknown": 0, 2021-05-12T12:07:14.264 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osd": 3, 2021-05-12T12:07:14.264 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_sum": { 2021-05-12T12:07:14.264 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:07:14.265 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48300, 2021-05-12T12:07:14.265 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:07:14.265 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:07:14.265 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:07:14.265 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:07:14.266 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:07:14.266 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:07:14.266 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:07:14.266 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:07:14.266 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:07:14.267 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:07:14.267 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:07:14.267 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:07:14.267 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 100, 2021-05-12T12:07:14.267 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 100, 2021-05-12T12:07:14.268 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:07:14.268 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:07:14.268 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:07:14.268 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:07:14.268 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:07:14.269 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:07:14.269 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:07:14.269 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:07:14.269 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:07:14.269 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:07:14.270 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:07:14.270 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:07:14.270 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:07:14.270 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:07:14.270 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:07:14.271 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:07:14.271 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:07:14.271 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:07:14.271 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:07:14.272 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:07:14.272 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:07:14.272 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:07:14.272 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:07:14.272 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:07:14.273 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:07:14.273 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.273 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:07:14.273 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:07:14.273 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:07:14.274 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:07:14.274 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 0, 2021-05-12T12:07:14.274 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 0, 2021-05-12T12:07:14.274 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:07:14.274 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:14.275 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:07:14.275 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:07:14.275 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:07:14.275 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.275 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 2048, 2021-05-12T12:07:14.276 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 2048, 2021-05-12T12:07:14.276 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 256, 2021-05-12T12:07:14.276 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 256, 2021-05-12T12:07:14.276 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 0 2021-05-12T12:07:14.276 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.277 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum": { 2021-05-12T12:07:14.277 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:07:14.277 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:07:14.277 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 257, 2021-05-12T12:07:14.277 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:07:14.278 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:07:14.278 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:07:14.278 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:07:14.278 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 24428, 2021-05-12T12:07:14.279 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2260, 2021-05-12T12:07:14.279 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:07:14.279 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 22141, 2021-05-12T12:07:14.279 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314548372, 2021-05-12T12:07:14.279 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:07:14.280 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:07:14.280 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322097532928, 2021-05-12T12:07:14.280 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:07:14.280 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2314240, 2021-05-12T12:07:14.280 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 502597, 2021-05-12T12:07:14.281 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:07:14.281 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:14.281 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:07:14.281 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:07:14.281 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 22673296 2021-05-12T12:07:14.282 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.282 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:07:14.282 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:07:14.282 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:07:14.282 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:07:14.283 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:07:14.283 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:07:14.283 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:07:14.283 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.283 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:07:14.284 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 92, 2021-05-12T12:07:14.284 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 92, 2021-05-12T12:07:14.284 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 92000000, 2021-05-12T12:07:14.284 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 92000000 2021-05-12T12:07:14.285 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.285 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:07:14.285 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:07:14.285 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.285 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum_by_class": { 2021-05-12T12:07:14.286 INFO:tasks.workunit.client.0.smithi191.stdout: "hdd": { 2021-05-12T12:07:14.286 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:07:14.286 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:07:14.286 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 257, 2021-05-12T12:07:14.286 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:07:14.287 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:07:14.287 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:07:14.287 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:07:14.287 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 24428, 2021-05-12T12:07:14.288 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2260, 2021-05-12T12:07:14.288 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:07:14.288 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 22141, 2021-05-12T12:07:14.288 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314548372, 2021-05-12T12:07:14.288 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:07:14.289 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:07:14.289 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322097532928, 2021-05-12T12:07:14.289 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:07:14.289 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2314240, 2021-05-12T12:07:14.289 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 502597, 2021-05-12T12:07:14.290 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:07:14.290 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:14.290 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:07:14.290 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:07:14.290 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 22673296 2021-05-12T12:07:14.291 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.291 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:07:14.291 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:07:14.291 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:07:14.292 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:07:14.292 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:07:14.292 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:07:14.292 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:07:14.292 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.293 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:07:14.293 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 92, 2021-05-12T12:07:14.293 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 92, 2021-05-12T12:07:14.293 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 92000000, 2021-05-12T12:07:14.293 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 92000000 2021-05-12T12:07:14.294 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.294 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:07:14.294 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:07:14.294 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.294 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.295 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_stats": [ 2021-05-12T12:07:14.295 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.295 INFO:tasks.workunit.client.0.smithi191.stdout: "poolid": 1, 2021-05-12T12:07:14.295 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 128, 2021-05-12T12:07:14.296 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:07:14.296 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48300, 2021-05-12T12:07:14.296 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:07:14.296 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:07:14.296 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:07:14.297 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:07:14.297 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:07:14.297 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:07:14.297 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:07:14.297 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:07:14.298 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:07:14.298 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:07:14.298 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:07:14.298 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:07:14.298 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 100, 2021-05-12T12:07:14.299 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 100, 2021-05-12T12:07:14.299 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:07:14.299 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:07:14.299 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:07:14.299 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:07:14.300 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:07:14.300 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:07:14.300 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:07:14.300 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:07:14.300 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:07:14.301 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:07:14.301 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:07:14.301 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:07:14.301 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:07:14.302 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:07:14.302 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:07:14.302 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:07:14.302 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:07:14.302 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:07:14.303 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:07:14.303 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:07:14.303 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:07:14.303 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:07:14.303 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:07:14.304 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:07:14.304 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:07:14.304 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.304 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:07:14.304 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:07:14.305 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:07:14.305 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:07:14.305 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 1220608, 2021-05-12T12:07:14.305 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 143228, 2021-05-12T12:07:14.305 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:07:14.306 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:14.306 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:07:14.306 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:07:14.306 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:07:14.306 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.307 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 2048, 2021-05-12T12:07:14.307 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 2048, 2021-05-12T12:07:14.307 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 256, 2021-05-12T12:07:14.307 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 256, 2021-05-12T12:07:14.307 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 3 2021-05-12T12:07:14.308 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.308 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.308 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_stats": [ 2021-05-12T12:07:14.308 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.309 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:07:14.309 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 176093659147 2021-05-12T12:07:14.309 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.309 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.309 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:07:14.310 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 42949672977 2021-05-12T12:07:14.310 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.310 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.310 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:07:14.310 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 184683593738 2021-05-12T12:07:14.311 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.311 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.311 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_state": [ 2021-05-12T12:07:14.311 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.311 INFO:tasks.workunit.client.0.smithi191.stdout: "state": "active+undersized+degraded", 2021-05-12T12:07:14.312 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 100 2021-05-12T12:07:14.312 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.312 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.312 INFO:tasks.workunit.client.0.smithi191.stdout: "state": "active+undersized", 2021-05-12T12:07:14.312 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 28 2021-05-12T12:07:14.313 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.313 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.313 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_osd": [ 2021-05-12T12:07:14.313 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.314 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:07:14.314 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:07:14.314 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 128, 2021-05-12T12:07:14.314 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:07:14.314 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.315 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.315 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:07:14.315 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:07:14.315 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 0, 2021-05-12T12:07:14.315 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:07:14.316 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.316 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.316 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:07:14.316 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 128, 2021-05-12T12:07:14.317 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 128, 2021-05-12T12:07:14.317 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:07:14.317 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.317 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.317 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": [ 2021-05-12T12:07:14.318 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:14.318 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:07:14.318 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": {} 2021-05-12T12:07:14.318 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.318 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:14.319 INFO:tasks.workunit.client.0.smithi191.stdout: "servicemap": { 2021-05-12T12:07:14.319 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 4, 2021-05-12T12:07:14.319 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:06:30.216324+0000", 2021-05-12T12:07:14.319 INFO:tasks.workunit.client.0.smithi191.stdout: "services": { 2021-05-12T12:07:14.319 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": { 2021-05-12T12:07:14.320 INFO:tasks.workunit.client.0.smithi191.stdout: "daemons": { 2021-05-12T12:07:14.320 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": "", 2021-05-12T12:07:14.320 INFO:tasks.workunit.client.0.smithi191.stdout: "0": { 2021-05-12T12:07:14.320 INFO:tasks.workunit.client.0.smithi191.stdout: "start_epoch": 0, 2021-05-12T12:07:14.320 INFO:tasks.workunit.client.0.smithi191.stdout: "start_stamp": "0.000000", 2021-05-12T12:07:14.321 INFO:tasks.workunit.client.0.smithi191.stdout: "gid": 0, 2021-05-12T12:07:14.321 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:07:14.321 INFO:tasks.workunit.client.0.smithi191.stdout: "metadata": {}, 2021-05-12T12:07:14.321 INFO:tasks.workunit.client.0.smithi191.stdout: "task_status": {} 2021-05-12T12:07:14.322 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.322 INFO:tasks.workunit.client.0.smithi191.stdout: "2": { 2021-05-12T12:07:14.322 INFO:tasks.workunit.client.0.smithi191.stdout: "start_epoch": 0, 2021-05-12T12:07:14.322 INFO:tasks.workunit.client.0.smithi191.stdout: "start_stamp": "0.000000", 2021-05-12T12:07:14.322 INFO:tasks.workunit.client.0.smithi191.stdout: "gid": 0, 2021-05-12T12:07:14.323 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:07:14.323 INFO:tasks.workunit.client.0.smithi191.stdout: "metadata": {}, 2021-05-12T12:07:14.323 INFO:tasks.workunit.client.0.smithi191.stdout: "task_status": {} 2021-05-12T12:07:14.323 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.323 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.324 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.324 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.324 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:14.324 INFO:tasks.workunit.client.0.smithi191.stdout: "paxos": { 2021-05-12T12:07:14.324 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:07:14.325 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 396, 2021-05-12T12:07:14.325 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pn": 0, 2021-05-12T12:07:14.325 INFO:tasks.workunit.client.0.smithi191.stdout: "accepted_pn": 0 2021-05-12T12:07:14.325 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:14.325 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:07:14.327 INFO:tasks.workunit.client.0.smithi191.stderr:report 1322483579 2021-05-12T12:07:14.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:07:14.327 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2021-05-12T12:07:14.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2021-05-12T12:07:14.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2021-05-12T12:07:14.328 INFO:tasks.workunit.client.0.smithi191.stdout:writing non-divergent object existing_4 2021-05-12T12:07:14.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:07:14.387 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:07:14.387 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.946402+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.388 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941992+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.388 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942552+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.388 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050555+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.388 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049600+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.389 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846594+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.389 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049564+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.389 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846817+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.389 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.051030+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.389 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942290+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.390 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945780+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.390 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.847177+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.390 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942200+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.390 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833041+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.391 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947498+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.391 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833088+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.391 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942729+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.391 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.832664+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.391 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050869+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.392 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.837911+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.392 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945483+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.392 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833729+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.392 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941846+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.393 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833578+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.393 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947161+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.393 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833814+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.393 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942362+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.393 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833464+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.394 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050707+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.394 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840401+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.394 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945174+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.394 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839289+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.394 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.395 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942000+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.395 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942817+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.395 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840197+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.395 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941588+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838547+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050396+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843504+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846242+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.396 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841485+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846461+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943327+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941935+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.397 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842661+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.398 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846589+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.398 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842451+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.398 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832706+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.398 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838298+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.398 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837624+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945419+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842097+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844805+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839883+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.399 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844828+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.400 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942564+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.400 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941545+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.400 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841976+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.400 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845732+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840522+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050125+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844172+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845879+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.401 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842251+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.402 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846106+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.402 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.944548+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.402 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.847030+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.402 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843740+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845368+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845354+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945126+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845891+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.403 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844452+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.844480+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844120+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845162+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.405 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845398+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.405 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945706+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.405 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844119+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.405 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842974+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.405 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844093+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844900+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.944832+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844564+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.406 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839359+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.407 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846643+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.407 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941056+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.407 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845734+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.407 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838171+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.407 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845534+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.408 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839523+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.408 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945992+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.408 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842066+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.408 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842614+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842320+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844527+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943028+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842817+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.409 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840183+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842977+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842824+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943625+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838009+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.410 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841875+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.411 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839833+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.411 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843815+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.411 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.411 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839918+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.411 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838521+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841628+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839901+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942281+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833094+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.412 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839602+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.413 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833452+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.413 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841714+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.413 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833276+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.413 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833821+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833336+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839029+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837516+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837177+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.414 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.715297+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.415 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837775+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.415 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832674+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:14.415 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:07:14.415 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:07:14.416 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:07:14.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T12:07:14.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:07:14.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2021-05-12T12:07:14.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:07:14.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:07:14.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:07:14.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:07:14.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:07:14.562 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:07:14.562 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:07:14.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:07:14.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:07:14.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:07:14.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:07:14.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:07:14.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:07:14.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:07:14.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:07:15.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:07:15.005 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:07:15.006 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:07:15.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:07:15.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:07:15.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:07:15.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=176093659150 2021-05-12T12:07:15.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 176093659150 2021-05-12T12:07:15.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-176093659150' 2021-05-12T12:07:15.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:07:15.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:07:15.260 INFO:tasks.workunit.client.0.smithi191.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:07:15.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:07:15.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:07:15.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:07:15.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:07:15.262 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:07:15.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=184683593741 2021-05-12T12:07:15.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 184683593741 2021-05-12T12:07:15.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-176093659150 2-184683593741' 2021-05-12T12:07:15.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:07:15.397 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-176093659150 2021-05-12T12:07:15.397 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:07:15.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:07:15.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-176093659150 2021-05-12T12:07:15.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:07:15.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=176093659150 2021-05-12T12:07:15.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 176093659150' 2021-05-12T12:07:15.400 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 176093659150 2021-05-12T12:07:15.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:07:15.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659147 -lt 176093659150 2021-05-12T12:07:15.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:07:16.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:07:16.765 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:07:17.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659150 -lt 176093659150 2021-05-12T12:07:17.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:07:17.138 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-184683593741 2021-05-12T12:07:17.138 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:07:17.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:07:17.139 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-184683593741 2021-05-12T12:07:17.139 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:07:17.140 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=184683593741 2021-05-12T12:07:17.140 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 184683593741' 2021-05-12T12:07:17.141 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 184683593741 2021-05-12T12:07:17.141 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:07:17.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593741 -lt 184683593741 2021-05-12T12:07:17.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:07:17.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:17.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:18.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 128 == 0 2021-05-12T12:07:18.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:18.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:18.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:18.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:18.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:18.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:18.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:18.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:18.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:18.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:18.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:18.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:18.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:07:18.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:07:18.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:07:18.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:18.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:07:18.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:18.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:18.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:18.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:18.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:18.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:18.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:18.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:19.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:19.346 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:19.346 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:19.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:19.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:19.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:19.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:19.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:19.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:19.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:19.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:19.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:19.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:20.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:20.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:20.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:07:20.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:20.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:07:20.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:20.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:20.537 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:20.537 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:20.537 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:20.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:20.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:20.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:20.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:20.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:20.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:20.912 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:21.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:21.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:21.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:21.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:21.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:21.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:21.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:21.391 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:21.391 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:21.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:21.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:21.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:07:21.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:21.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:07:22.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:22.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:22.275 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:22.275 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:22.275 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:22.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:22.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:22.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:22.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:22.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:22.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:22.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:23.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:23.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:23.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:23.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:23.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:23.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:23.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:23.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:23.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:23.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:23.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:23.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:07:23.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:23.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:07:24.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:24.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:24.475 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:24.475 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:24.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:24.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:24.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:24.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:24.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:24.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:24.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:24.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:25.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:25.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:25.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:25.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:25.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:25.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:25.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:25.340 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:25.340 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:25.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:25.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:25.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:07:25.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:25.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:07:27.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:27.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:27.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:27.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:27.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:27.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:27.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:27.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:27.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:27.803 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:27.803 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:27.804 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:28.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:28.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:28.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:28.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:28.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:28.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:28.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:28.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:28.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:28.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:28.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:28.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:07:28.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:28.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:07:31.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:31.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:31.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:31.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:31.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:31.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:31.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:31.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:32.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:32.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:32.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:32.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:32.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:32.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:32.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:32.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:32.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:32.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:32.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:32.902 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:32.902 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:33.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:33.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:33.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:07:33.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:33.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:07:39.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:39.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:39.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:39.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:39.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:39.782 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:39.782 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:39.782 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:40.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:40.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:40.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:40.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:40.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:40.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:40.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:40.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:40.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:40.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:40.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:40.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:40.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:41.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:41.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:41.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:07:41.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:41.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:07:48.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:48.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:48.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:48.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:48.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:48.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:48.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:48.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:48.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:48.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:48.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:48.816 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:49.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:49.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:49.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:49.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:49.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:49.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:49.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:49.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:49.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:49.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:49.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:49.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:07:49.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:07:50.247 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:07:50.248 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_fingerprint": "43ec61a6-6986-4458-bee9-93f5dd8a4db2", 2021-05-12T12:07:50.248 INFO:tasks.workunit.client.0.smithi191.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:50.248 INFO:tasks.workunit.client.0.smithi191.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:07:50.249 INFO:tasks.workunit.client.0.smithi191.stdout: "timestamp": "2021-05-12T12:07:50.240276+0000", 2021-05-12T12:07:50.249 INFO:tasks.workunit.client.0.smithi191.stdout: "tag": "", 2021-05-12T12:07:50.249 INFO:tasks.workunit.client.0.smithi191.stdout: "health": { 2021-05-12T12:07:50.249 INFO:tasks.workunit.client.0.smithi191.stdout: "status": "HEALTH_ERR", 2021-05-12T12:07:50.250 INFO:tasks.workunit.client.0.smithi191.stdout: "checks": { 2021-05-12T12:07:50.250 INFO:tasks.workunit.client.0.smithi191.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:07:50.251 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:07:50.251 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:50.251 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:07:50.251 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:07:50.252 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.252 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:07:50.253 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.253 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:07:50.253 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.254 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.254 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:50.254 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.254 INFO:tasks.workunit.client.0.smithi191.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:07:50.255 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:50.255 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:50.255 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:07:50.255 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 17 2021-05-12T12:07:50.256 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.256 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [], 2021-05-12T12:07:50.256 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:50.256 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.257 INFO:tasks.workunit.client.0.smithi191.stdout: "OSD_DOWN": { 2021-05-12T12:07:50.257 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:50.257 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:50.258 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 osds down", 2021-05-12T12:07:50.258 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:07:50.258 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.259 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:07:50.259 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.259 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "osd.1 (root=default,host=smithi191) is down" 2021-05-12T12:07:50.259 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.260 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.260 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:50.260 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.261 INFO:tasks.workunit.client.0.smithi191.stdout: "PG_DEGRADED": { 2021-05-12T12:07:50.261 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:50.261 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:50.262 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 100 pgs degraded, 128 pgs undersized", 2021-05-12T12:07:50.262 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 228 2021-05-12T12:07:50.263 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.263 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:07:50.263 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.264 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.23 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:50.264 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.265 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.265 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.4e is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.266 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.266 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.266 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.4f is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.267 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.267 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.267 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.50 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.268 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.269 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.269 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.51 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.269 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.270 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.270 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.52 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.271 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.271 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.272 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.53 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.272 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.272 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.273 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.54 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.273 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.274 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.274 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.55 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.275 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.276 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.276 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.56 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.276 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.277 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.277 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.57 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.278 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.278 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.279 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.58 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.279 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.279 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.280 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.59 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.280 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.280 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.281 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5a is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.281 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.282 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.282 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5b is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.283 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.283 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.283 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5c is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.284 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.284 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.285 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5d is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.285 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.285 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.286 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5e is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.286 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.286 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.287 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.5f is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.288 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.288 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.288 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.60 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.289 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.289 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.290 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.61 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.290 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.290 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.291 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.62 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.291 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.292 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.292 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.63 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.292 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.293 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.293 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.64 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:07:50.293 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.294 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.294 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.65 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.294 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.295 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.295 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.66 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.295 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.296 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.296 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.67 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.296 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.297 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.297 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.68 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.297 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.297 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.298 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.69 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.298 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.298 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.299 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.6a is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.299 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.299 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.299 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.6b is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.300 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.300 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.300 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.6c is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.300 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.301 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.301 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.6d is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.301 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.302 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.302 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.6e is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.302 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.302 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.303 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.6f is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.303 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.303 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.303 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.70 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.304 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.304 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.304 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.71 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.305 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.305 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.305 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.72 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.306 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.306 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.306 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.73 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.306 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.307 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.307 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.74 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.307 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.307 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.308 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.75 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.308 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.308 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.309 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.76 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.309 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.309 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.310 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.77 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.310 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.310 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.310 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.78 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.311 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.311 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.311 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.79 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.312 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.312 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.312 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.7a is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.312 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.313 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.313 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.7b is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.313 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.313 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.314 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.7c is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.314 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.314 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.314 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.7d is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.315 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.315 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.315 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.7e is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.316 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.316 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.316 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.7f is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:07:50.316 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.317 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.317 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:50.317 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.317 INFO:tasks.workunit.client.0.smithi191.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:07:50.318 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:50.318 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:07:50.318 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:07:50.318 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:07:50.319 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.319 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:07:50.319 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.319 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:07:50.319 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.320 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.320 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:07:50.320 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.320 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.321 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:07:50.321 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.321 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.321 INFO:tasks.workunit.client.0.smithi191.stdout: "mutes": [] 2021-05-12T12:07:50.322 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.322 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_first_committed": 1, 2021-05-12T12:07:50.322 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_last_committed": 1, 2021-05-12T12:07:50.322 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap": { 2021-05-12T12:07:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:07:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:07:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:05:07.934785Z", 2021-05-12T12:07:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:05:07.934785Z", 2021-05-12T12:07:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release": 17, 2021-05-12T12:07:50.324 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:07:50.324 INFO:tasks.workunit.client.0.smithi191.stdout: "election_strategy": 1, 2021-05-12T12:07:50.324 INFO:tasks.workunit.client.0.smithi191.stdout: "disallowed_leaders: ": "", 2021-05-12T12:07:50.324 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": false, 2021-05-12T12:07:50.325 INFO:tasks.workunit.client.0.smithi191.stdout: "features": { 2021-05-12T12:07:50.325 INFO:tasks.workunit.client.0.smithi191.stdout: "persistent": [ 2021-05-12T12:07:50.325 INFO:tasks.workunit.client.0.smithi191.stdout: "kraken", 2021-05-12T12:07:50.325 INFO:tasks.workunit.client.0.smithi191.stdout: "luminous", 2021-05-12T12:07:50.326 INFO:tasks.workunit.client.0.smithi191.stdout: "mimic", 2021-05-12T12:07:50.326 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap-prune", 2021-05-12T12:07:50.326 INFO:tasks.workunit.client.0.smithi191.stdout: "nautilus", 2021-05-12T12:07:50.326 INFO:tasks.workunit.client.0.smithi191.stdout: "octopus", 2021-05-12T12:07:50.327 INFO:tasks.workunit.client.0.smithi191.stdout: "pacific", 2021-05-12T12:07:50.327 INFO:tasks.workunit.client.0.smithi191.stdout: "elector-pinging", 2021-05-12T12:07:50.327 INFO:tasks.workunit.client.0.smithi191.stdout: "quincy" 2021-05-12T12:07:50.327 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.328 INFO:tasks.workunit.client.0.smithi191.stdout: "optional": [] 2021-05-12T12:07:50.328 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.328 INFO:tasks.workunit.client.0.smithi191.stdout: "mons": [ 2021-05-12T12:07:50.328 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.329 INFO:tasks.workunit.client.0.smithi191.stdout: "rank": 0, 2021-05-12T12:07:50.329 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "a", 2021-05-12T12:07:50.329 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:07:50.329 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.329 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.330 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.330 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:07:50.330 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 0 2021-05-12T12:07:50.330 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.331 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.331 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.331 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:07:50.331 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:07:50.332 INFO:tasks.workunit.client.0.smithi191.stdout: "priority": 0, 2021-05-12T12:07:50.332 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 0, 2021-05-12T12:07:50.332 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_location": "{}" 2021-05-12T12:07:50.332 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.333 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.333 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.333 INFO:tasks.workunit.client.0.smithi191.stdout: "quorum": [ 2021-05-12T12:07:50.333 INFO:tasks.workunit.client.0.smithi191.stdout: 0 2021-05-12T12:07:50.333 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.334 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap": { 2021-05-12T12:07:50.334 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 48, 2021-05-12T12:07:50.334 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:07:50.334 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:05:08.774093+0000", 2021-05-12T12:07:50.335 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:06:38.214829+0000", 2021-05-12T12:07:50.335 INFO:tasks.workunit.client.0.smithi191.stdout: "last_up_change": "2021-05-12T12:06:35.204550+0000", 2021-05-12T12:07:50.335 INFO:tasks.workunit.client.0.smithi191.stdout: "last_in_change": "2021-05-12T12:06:37.743635+0000", 2021-05-12T12:07:50.335 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:07:50.335 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_num": 5800384, 2021-05-12T12:07:50.336 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_set": [ 2021-05-12T12:07:50.336 INFO:tasks.workunit.client.0.smithi191.stdout: "nodown", 2021-05-12T12:07:50.336 INFO:tasks.workunit.client.0.smithi191.stdout: "noin", 2021-05-12T12:07:50.336 INFO:tasks.workunit.client.0.smithi191.stdout: "noout", 2021-05-12T12:07:50.336 INFO:tasks.workunit.client.0.smithi191.stdout: "pglog_hardlimit", 2021-05-12T12:07:50.337 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snapdirs", 2021-05-12T12:07:50.337 INFO:tasks.workunit.client.0.smithi191.stdout: "recovery_deletes", 2021-05-12T12:07:50.337 INFO:tasks.workunit.client.0.smithi191.stdout: "sortbitwise" 2021-05-12T12:07:50.337 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.337 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_version": 9, 2021-05-12T12:07:50.338 INFO:tasks.workunit.client.0.smithi191.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:07:50.338 INFO:tasks.workunit.client.0.smithi191.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:07:50.338 INFO:tasks.workunit.client.0.smithi191.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:07:50.338 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_snapshot": "", 2021-05-12T12:07:50.338 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_max": 1, 2021-05-12T12:07:50.339 INFO:tasks.workunit.client.0.smithi191.stdout: "max_osd": 3, 2021-05-12T12:07:50.339 INFO:tasks.workunit.client.0.smithi191.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:07:50.339 INFO:tasks.workunit.client.0.smithi191.stdout: "min_compat_client": "jewel", 2021-05-12T12:07:50.339 INFO:tasks.workunit.client.0.smithi191.stdout: "require_osd_release": "quincy", 2021-05-12T12:07:50.340 INFO:tasks.workunit.client.0.smithi191.stdout: "pools": [ 2021-05-12T12:07:50.340 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.340 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:07:50.340 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_name": "test", 2021-05-12T12:07:50.340 INFO:tasks.workunit.client.0.smithi191.stdout: "create_time": "2021-05-12T12:05:43.641277+0000", 2021-05-12T12:07:50.341 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": 1, 2021-05-12T12:07:50.341 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_names": "hashpspool", 2021-05-12T12:07:50.341 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:07:50.341 INFO:tasks.workunit.client.0.smithi191.stdout: "size": 3, 2021-05-12T12:07:50.341 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 2, 2021-05-12T12:07:50.342 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_rule": 0, 2021-05-12T12:07:50.342 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:07:50.342 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:07:50.342 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:07:50.342 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:07:50.343 INFO:tasks.workunit.client.0.smithi191.stdout: "object_hash": 2, 2021-05-12T12:07:50.343 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_autoscale_mode": "on", 2021-05-12T12:07:50.343 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num": 128, 2021-05-12T12:07:50.343 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num": 1, 2021-05-12T12:07:50.344 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num_target": 128, 2021-05-12T12:07:50.344 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_target": 128, 2021-05-12T12:07:50.344 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_pending": 128, 2021-05-12T12:07:50.344 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pg_merge_meta": { 2021-05-12T12:07:50.344 INFO:tasks.workunit.client.0.smithi191.stdout: "source_pgid": "0.0", 2021-05-12T12:07:50.345 INFO:tasks.workunit.client.0.smithi191.stdout: "ready_epoch": 0, 2021-05-12T12:07:50.345 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_started": 0, 2021-05-12T12:07:50.345 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": 0, 2021-05-12T12:07:50.345 INFO:tasks.workunit.client.0.smithi191.stdout: "source_version": "0'0", 2021-05-12T12:07:50.345 INFO:tasks.workunit.client.0.smithi191.stdout: "target_version": "0'0" 2021-05-12T12:07:50.346 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.346 INFO:tasks.workunit.client.0.smithi191.stdout: "last_change": "33", 2021-05-12T12:07:50.346 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend": "0", 2021-05-12T12:07:50.346 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:07:50.347 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_preluminous": "33", 2021-05-12T12:07:50.347 INFO:tasks.workunit.client.0.smithi191.stdout: "auid": 0, 2021-05-12T12:07:50.347 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:07:50.347 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_seq": 0, 2021-05-12T12:07:50.347 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_epoch": 0, 2021-05-12T12:07:50.348 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_snaps": [], 2021-05-12T12:07:50.348 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps": "[]", 2021-05-12T12:07:50.348 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_bytes": 0, 2021-05-12T12:07:50.348 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_objects": 0, 2021-05-12T12:07:50.348 INFO:tasks.workunit.client.0.smithi191.stdout: "tiers": [], 2021-05-12T12:07:50.349 INFO:tasks.workunit.client.0.smithi191.stdout: "tier_of": -1, 2021-05-12T12:07:50.349 INFO:tasks.workunit.client.0.smithi191.stdout: "read_tier": -1, 2021-05-12T12:07:50.349 INFO:tasks.workunit.client.0.smithi191.stdout: "write_tier": -1, 2021-05-12T12:07:50.349 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_mode": "none", 2021-05-12T12:07:50.349 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_bytes": 0, 2021-05-12T12:07:50.350 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_objects": 0, 2021-05-12T12:07:50.350 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:07:50.350 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:07:50.350 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:07:50.351 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_flush_age": 0, 2021-05-12T12:07:50.351 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_evict_age": 0, 2021-05-12T12:07:50.351 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profile": "", 2021-05-12T12:07:50.351 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_params": { 2021-05-12T12:07:50.351 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "none" 2021-05-12T12:07:50.352 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.352 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_period": 0, 2021-05-12T12:07:50.352 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_count": 0, 2021-05-12T12:07:50.352 INFO:tasks.workunit.client.0.smithi191.stdout: "use_gmt_hitset": true, 2021-05-12T12:07:50.352 INFO:tasks.workunit.client.0.smithi191.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:07:50.353 INFO:tasks.workunit.client.0.smithi191.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:07:50.353 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:07:50.353 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:07:50.353 INFO:tasks.workunit.client.0.smithi191.stdout: "grade_table": [], 2021-05-12T12:07:50.354 INFO:tasks.workunit.client.0.smithi191.stdout: "stripe_width": 0, 2021-05-12T12:07:50.354 INFO:tasks.workunit.client.0.smithi191.stdout: "expected_num_objects": 0, 2021-05-12T12:07:50.354 INFO:tasks.workunit.client.0.smithi191.stdout: "fast_read": false, 2021-05-12T12:07:50.354 INFO:tasks.workunit.client.0.smithi191.stdout: "options": {}, 2021-05-12T12:07:50.354 INFO:tasks.workunit.client.0.smithi191.stdout: "application_metadata": {} 2021-05-12T12:07:50.355 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.355 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.355 INFO:tasks.workunit.client.0.smithi191.stdout: "osds": [ 2021-05-12T12:07:50.355 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.355 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:07:50.356 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "efb617d0-a38f-4017-94d3-fc68ac2ed9be", 2021-05-12T12:07:50.356 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:07:50.356 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:07:50.356 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:07:50.356 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:07:50.357 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 5, 2021-05-12T12:07:50.357 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 31, 2021-05-12T12:07:50.357 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 41, 2021-05-12T12:07:50.357 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 47, 2021-05-12T12:07:50.357 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 35, 2021-05-12T12:07:50.358 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:07:50.358 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:07:50.358 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.358 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.359 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.359 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:07:50.359 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:50.359 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.359 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.360 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.360 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:07:50.360 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:50.360 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.360 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.361 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.361 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:07:50.361 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.361 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.362 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.362 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:07:50.362 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:50.362 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.362 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.363 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.363 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:07:50.363 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:50.363 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.363 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.364 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.364 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:07:50.364 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.364 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.364 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.365 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:07:50.365 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:50.365 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.365 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.366 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.366 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:07:50.366 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:50.366 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.366 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.367 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.367 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:07:50.367 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.367 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.367 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.368 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:07:50.368 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:50.368 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.368 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.368 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.369 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:07:50.369 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 67634 2021-05-12T12:07:50.369 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.369 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.370 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.370 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6803/67634", 2021-05-12T12:07:50.370 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6805/67634", 2021-05-12T12:07:50.370 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6809/67634", 2021-05-12T12:07:50.370 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6807/67634", 2021-05-12T12:07:50.371 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:07:50.371 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:07:50.371 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:07:50.371 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.372 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.372 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.372 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:07:50.372 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "5fe20a5a-c3fb-42c5-aadd-2c4671a13114", 2021-05-12T12:07:50.372 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 0, 2021-05-12T12:07:50.373 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:07:50.373 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:07:50.373 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:07:50.373 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 0, 2021-05-12T12:07:50.373 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 0, 2021-05-12T12:07:50.374 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 10, 2021-05-12T12:07:50.374 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 33, 2021-05-12T12:07:50.374 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 36, 2021-05-12T12:07:50.374 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:07:50.374 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:07:50.375 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.375 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.375 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.375 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:07:50.375 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:50.376 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.376 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.376 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.376 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:07:50.377 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:50.377 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.377 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.377 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.377 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:07:50.378 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.378 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.378 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.378 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:07:50.378 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:50.379 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.379 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.379 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.379 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:07:50.380 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:50.380 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.380 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.380 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.380 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:07:50.381 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.381 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.381 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.381 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:07:50.381 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:50.382 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.382 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.382 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.382 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:07:50.383 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:50.383 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.383 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.383 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.383 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:07:50.384 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.384 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.384 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.384 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:07:50.384 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:50.385 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.385 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.385 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.385 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:07:50.385 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 62131 2021-05-12T12:07:50.386 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.386 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.386 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.386 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/62131", 2021-05-12T12:07:50.387 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/62131", 2021-05-12T12:07:50.387 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/62131", 2021-05-12T12:07:50.387 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/62131", 2021-05-12T12:07:50.387 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:07:50.387 INFO:tasks.workunit.client.0.smithi191.stdout: "exists" 2021-05-12T12:07:50.388 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.388 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.388 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.388 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:07:50.388 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "16b8b89a-4aa1-4296-88f5-d60bf3eabab1", 2021-05-12T12:07:50.389 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:07:50.389 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:07:50.389 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:07:50.389 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:07:50.389 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 15, 2021-05-12T12:07:50.390 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 31, 2021-05-12T12:07:50.390 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 43, 2021-05-12T12:07:50.390 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 0, 2021-05-12T12:07:50.390 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 37, 2021-05-12T12:07:50.390 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:07:50.391 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:07:50.391 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.391 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.391 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.391 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:07:50.392 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:50.392 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.392 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.392 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.393 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:07:50.393 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:50.393 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.393 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.393 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.394 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:07:50.394 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.394 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.394 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.395 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:07:50.395 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:50.395 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.396 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.396 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.396 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:07:50.396 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:50.397 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.397 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.397 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.397 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:07:50.397 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.398 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.398 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.398 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:07:50.398 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:50.399 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.399 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.399 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.399 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:07:50.399 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:50.400 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.400 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.400 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.400 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:07:50.400 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:07:50.401 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.401 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:07:50.401 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:07:50.401 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:50.401 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.402 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.402 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:07:50.402 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:07:50.402 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 68217 2021-05-12T12:07:50.403 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.403 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.403 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.403 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/68217", 2021-05-12T12:07:50.403 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/68217", 2021-05-12T12:07:50.404 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/68217", 2021-05-12T12:07:50.404 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/68217", 2021-05-12T12:07:50.404 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:07:50.404 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:07:50.404 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:07:50.405 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.405 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.405 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.405 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_xinfo": [ 2021-05-12T12:07:50.406 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.406 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:07:50.406 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:06:23.845446+0000", 2021-05-12T12:07:50.406 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:07:50.406 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:07:50.407 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:07:50.407 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:07:50.407 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:19.111982+0000", 2021-05-12T12:07:50.407 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:07:50.407 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.408 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.408 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:07:50.408 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:06:24.454727+0000", 2021-05-12T12:07:50.408 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:07:50.409 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:07:50.409 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:07:50.409 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:07:50.409 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:28.849900+0000", 2021-05-12T12:07:50.409 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:07:50.410 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.410 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.410 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:07:50.410 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:06:25.022069+0000", 2021-05-12T12:07:50.410 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:07:50.411 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:07:50.411 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:07:50.411 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:07:50.411 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:40.000052+0000", 2021-05-12T12:07:50.411 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:07:50.412 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.412 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.412 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap": [], 2021-05-12T12:07:50.412 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap_items": [], 2021-05-12T12:07:50.412 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_temp": [], 2021-05-12T12:07:50.413 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_temp": [], 2021-05-12T12:07:50.413 INFO:tasks.workunit.client.0.smithi191.stdout: "blocklist": {}, 2021-05-12T12:07:50.413 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profiles": { 2021-05-12T12:07:50.413 INFO:tasks.workunit.client.0.smithi191.stdout: "default": { 2021-05-12T12:07:50.414 INFO:tasks.workunit.client.0.smithi191.stdout: "k": "2", 2021-05-12T12:07:50.414 INFO:tasks.workunit.client.0.smithi191.stdout: "m": "2", 2021-05-12T12:07:50.414 INFO:tasks.workunit.client.0.smithi191.stdout: "plugin": "jerasure", 2021-05-12T12:07:50.414 INFO:tasks.workunit.client.0.smithi191.stdout: "technique": "reed_sol_van" 2021-05-12T12:07:50.414 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.415 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.415 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps_queue": [], 2021-05-12T12:07:50.415 INFO:tasks.workunit.client.0.smithi191.stdout: "new_removed_snaps": [], 2021-05-12T12:07:50.415 INFO:tasks.workunit.client.0.smithi191.stdout: "new_purged_snaps": [], 2021-05-12T12:07:50.415 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_node_flags": {}, 2021-05-12T12:07:50.416 INFO:tasks.workunit.client.0.smithi191.stdout: "device_class_flags": {}, 2021-05-12T12:07:50.416 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": { 2021-05-12T12:07:50.416 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_enabled": false, 2021-05-12T12:07:50.416 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_bucket_count": 0, 2021-05-12T12:07:50.416 INFO:tasks.workunit.client.0.smithi191.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:07:50.417 INFO:tasks.workunit.client.0.smithi191.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:07:50.417 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_bucket": 0 2021-05-12T12:07:50.417 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.417 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.418 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_metadata": [ 2021-05-12T12:07:50.418 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.418 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:50.418 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:07:50.418 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6804/67634,v1:127.0.0.1:6805/67634]", 2021-05-12T12:07:50.419 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:07:50.419 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:07:50.419 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:07:50.419 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:07:50.419 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:07:50.420 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:07:50.420 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:07:50.420 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:07:50.420 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:07:50.421 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:07:50.421 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:07:50.421 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:07:50.421 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:07:50.421 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:07:50.422 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:07:50.422 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:07:50.422 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:50.422 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:07:50.422 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:07:50.423 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:07:50.423 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:07:50.423 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:07:50.423 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:07:50.424 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:07:50.424 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:07:50.424 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6802/67634,v1:127.0.0.1:6803/67634]", 2021-05-12T12:07:50.424 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:07:50.424 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/67634,v1:127.0.0.1:6809/67634]", 2021-05-12T12:07:50.425 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/67634,v1:127.0.0.1:6807/67634]", 2021-05-12T12:07:50.425 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:07:50.425 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:07:50.425 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:07:50.426 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:07:50.426 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:07:50.426 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:07:50.426 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:07:50.426 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:07:50.427 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:07:50.427 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:07:50.427 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:07:50.427 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:07:50.427 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:07:50.428 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.428 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.428 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:50.428 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:07:50.429 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/62131,v1:127.0.0.1:6813/62131]", 2021-05-12T12:07:50.429 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:07:50.429 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:07:50.429 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:07:50.429 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:07:50.430 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:07:50.430 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:07:50.430 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:07:50.430 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:07:50.430 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:07:50.431 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:07:50.431 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:07:50.431 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:07:50.431 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:07:50.431 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:07:50.432 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:07:50.432 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:07:50.432 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:50.432 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:07:50.432 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:07:50.433 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:07:50.433 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:07:50.433 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:07:50.433 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:07:50.434 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:07:50.434 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:07:50.434 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/62131,v1:127.0.0.1:6811/62131]", 2021-05-12T12:07:50.434 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:07:50.434 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/62131,v1:127.0.0.1:6817/62131]", 2021-05-12T12:07:50.435 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/62131,v1:127.0.0.1:6815/62131]", 2021-05-12T12:07:50.435 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:07:50.435 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:07:50.435 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:07:50.435 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:07:50.436 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:07:50.436 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:07:50.436 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:07:50.436 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:07:50.436 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:07:50.437 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:07:50.437 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:07:50.437 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:07:50.437 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:07:50.438 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.438 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.438 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:50.438 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:07:50.438 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/68217,v1:127.0.0.1:6813/68217]", 2021-05-12T12:07:50.439 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:07:50.439 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:07:50.439 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:07:50.439 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:07:50.439 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:07:50.440 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:07:50.440 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:07:50.440 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:07:50.440 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:07:50.440 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:07:50.441 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:07:50.441 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:07:50.441 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:07:50.441 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:07:50.441 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:07:50.442 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:07:50.442 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:50.442 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:07:50.442 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:07:50.443 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:07:50.444 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:07:50.444 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:07:50.444 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:07:50.444 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:07:50.445 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:07:50.445 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/68217,v1:127.0.0.1:6811/68217]", 2021-05-12T12:07:50.445 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:07:50.445 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/68217,v1:127.0.0.1:6817/68217]", 2021-05-12T12:07:50.445 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/68217,v1:127.0.0.1:6815/68217]", 2021-05-12T12:07:50.446 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:07:50.446 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:07:50.446 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:07:50.446 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:07:50.447 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:07:50.447 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:07:50.447 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:07:50.447 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:07:50.447 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:07:50.448 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:07:50.448 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:07:50.448 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:07:50.448 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:07:50.449 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.449 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.449 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_clean_epochs": { 2021-05-12T12:07:50.449 INFO:tasks.workunit.client.0.smithi191.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:07:50.449 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": { 2021-05-12T12:07:50.450 INFO:tasks.workunit.client.0.smithi191.stdout: "per_pool": [] 2021-05-12T12:07:50.450 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.450 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_epochs": [ 2021-05-12T12:07:50.450 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.451 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:50.451 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 41 2021-05-12T12:07:50.451 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.451 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.451 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:50.452 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 10 2021-05-12T12:07:50.452 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.452 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.452 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:50.452 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 43 2021-05-12T12:07:50.453 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.453 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.453 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.453 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_first_committed": 1, 2021-05-12T12:07:50.453 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_last_committed": 48, 2021-05-12T12:07:50.454 INFO:tasks.workunit.client.0.smithi191.stdout: "crushmap": { 2021-05-12T12:07:50.454 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": [ 2021-05-12T12:07:50.454 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.454 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:50.455 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.0", 2021-05-12T12:07:50.455 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:07:50.455 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.455 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.455 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:50.456 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.1", 2021-05-12T12:07:50.456 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:07:50.456 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.456 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.456 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:50.457 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.2", 2021-05-12T12:07:50.457 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:07:50.457 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.457 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.457 INFO:tasks.workunit.client.0.smithi191.stdout: "types": [ 2021-05-12T12:07:50.458 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.458 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 0, 2021-05-12T12:07:50.458 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd" 2021-05-12T12:07:50.458 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.459 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.459 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:07:50.459 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "host" 2021-05-12T12:07:50.459 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.459 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.460 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 2, 2021-05-12T12:07:50.460 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "chassis" 2021-05-12T12:07:50.460 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.460 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.460 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 3, 2021-05-12T12:07:50.461 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "rack" 2021-05-12T12:07:50.461 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.461 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.461 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 4, 2021-05-12T12:07:50.461 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "row" 2021-05-12T12:07:50.462 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.462 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.462 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 5, 2021-05-12T12:07:50.462 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pdu" 2021-05-12T12:07:50.463 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.463 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.463 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 6, 2021-05-12T12:07:50.463 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pod" 2021-05-12T12:07:50.463 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.464 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.464 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 7, 2021-05-12T12:07:50.464 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "room" 2021-05-12T12:07:50.464 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.464 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.465 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 8, 2021-05-12T12:07:50.465 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "datacenter" 2021-05-12T12:07:50.465 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.465 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.465 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 9, 2021-05-12T12:07:50.466 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "zone" 2021-05-12T12:07:50.466 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.466 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.466 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 10, 2021-05-12T12:07:50.466 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "region" 2021-05-12T12:07:50.467 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.467 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.467 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:07:50.467 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "root" 2021-05-12T12:07:50.468 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.468 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.468 INFO:tasks.workunit.client.0.smithi191.stdout: "buckets": [ 2021-05-12T12:07:50.468 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.468 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -1, 2021-05-12T12:07:50.469 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default", 2021-05-12T12:07:50.469 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:07:50.469 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:07:50.469 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:50.469 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:07:50.470 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:07:50.470 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:07:50.470 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.470 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:07:50.471 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:50.471 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:07:50.471 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.471 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.471 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.472 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.472 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -2, 2021-05-12T12:07:50.472 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default~hdd", 2021-05-12T12:07:50.472 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:07:50.472 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:07:50.473 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:50.473 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:07:50.473 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:07:50.473 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:07:50.473 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.474 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:07:50.474 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:50.474 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:07:50.474 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.475 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.475 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.475 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.475 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:07:50.475 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191", 2021-05-12T12:07:50.476 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:07:50.476 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:07:50.476 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:50.476 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:07:50.476 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:07:50.477 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:07:50.477 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.477 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:50.477 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:50.478 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:07:50.478 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.478 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.478 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:50.478 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:50.479 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:07:50.479 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.479 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.479 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:50.479 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:50.480 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:07:50.480 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.480 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.480 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.480 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.481 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:07:50.481 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191~hdd", 2021-05-12T12:07:50.481 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:07:50.481 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:07:50.482 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:07:50.482 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:07:50.482 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:07:50.482 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:07:50.482 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.483 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:07:50.483 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:50.483 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:07:50.483 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.484 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.484 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:07:50.484 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:50.484 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:07:50.484 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.485 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.485 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:07:50.485 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:07:50.485 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:07:50.485 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.486 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.486 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.486 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.486 INFO:tasks.workunit.client.0.smithi191.stdout: "rules": [ 2021-05-12T12:07:50.486 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.487 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 0, 2021-05-12T12:07:50.487 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "replicated_rule", 2021-05-12T12:07:50.487 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 0, 2021-05-12T12:07:50.487 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:07:50.488 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 1, 2021-05-12T12:07:50.488 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 10, 2021-05-12T12:07:50.488 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2021-05-12T12:07:50.488 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.488 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2021-05-12T12:07:50.489 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2021-05-12T12:07:50.489 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2021-05-12T12:07:50.489 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.489 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.489 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_firstn", 2021-05-12T12:07:50.490 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2021-05-12T12:07:50.490 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2021-05-12T12:07:50.490 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.490 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.491 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2021-05-12T12:07:50.491 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.491 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:07:50.491 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.491 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.492 INFO:tasks.workunit.client.0.smithi191.stdout: "tunables": { 2021-05-12T12:07:50.492 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_tries": 0, 2021-05-12T12:07:50.492 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:07:50.492 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_total_tries": 50, 2021-05-12T12:07:50.492 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:07:50.493 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:07:50.493 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_stable": 1, 2021-05-12T12:07:50.493 INFO:tasks.workunit.client.0.smithi191.stdout: "straw_calc_version": 1, 2021-05-12T12:07:50.493 INFO:tasks.workunit.client.0.smithi191.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:07:50.494 INFO:tasks.workunit.client.0.smithi191.stdout: "profile": "jewel", 2021-05-12T12:07:50.494 INFO:tasks.workunit.client.0.smithi191.stdout: "optimal_tunables": 1, 2021-05-12T12:07:50.494 INFO:tasks.workunit.client.0.smithi191.stdout: "legacy_tunables": 0, 2021-05-12T12:07:50.494 INFO:tasks.workunit.client.0.smithi191.stdout: "minimum_required_version": "jewel", 2021-05-12T12:07:50.494 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables": 1, 2021-05-12T12:07:50.495 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables2": 1, 2021-05-12T12:07:50.495 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v2_rules": 0, 2021-05-12T12:07:50.495 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables3": 1, 2021-05-12T12:07:50.495 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v3_rules": 0, 2021-05-12T12:07:50.496 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v4_buckets": 1, 2021-05-12T12:07:50.496 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables5": 1, 2021-05-12T12:07:50.496 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v5_rules": 0 2021-05-12T12:07:50.496 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.496 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_args": {} 2021-05-12T12:07:50.496 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.497 INFO:tasks.workunit.client.0.smithi191.stdout: "fsmap": { 2021-05-12T12:07:50.497 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:07:50.497 INFO:tasks.workunit.client.0.smithi191.stdout: "default_fscid": -1, 2021-05-12T12:07:50.497 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": { 2021-05-12T12:07:50.498 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": {}, 2021-05-12T12:07:50.498 INFO:tasks.workunit.client.0.smithi191.stdout: "ro_compat": {}, 2021-05-12T12:07:50.498 INFO:tasks.workunit.client.0.smithi191.stdout: "incompat": { 2021-05-12T12:07:50.498 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_1": "base v0.20", 2021-05-12T12:07:50.498 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:07:50.499 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:07:50.499 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:07:50.499 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:07:50.499 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:07:50.500 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_8": "no anchor table", 2021-05-12T12:07:50.500 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_9": "file layout v2", 2021-05-12T12:07:50.500 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:07:50.500 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.500 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.501 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_flags": { 2021-05-12T12:07:50.501 INFO:tasks.workunit.client.0.smithi191.stdout: "enable_multiple": true, 2021-05-12T12:07:50.501 INFO:tasks.workunit.client.0.smithi191.stdout: "ever_enabled_multiple": true 2021-05-12T12:07:50.501 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.501 INFO:tasks.workunit.client.0.smithi191.stdout: "standbys": [], 2021-05-12T12:07:50.502 INFO:tasks.workunit.client.0.smithi191.stdout: "filesystems": [] 2021-05-12T12:07:50.502 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.502 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:07:50.502 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:07:50.502 INFO:tasks.workunit.client.0.smithi191.stdout: "auth": { 2021-05-12T12:07:50.503 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:07:50.503 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 5, 2021-05-12T12:07:50.503 INFO:tasks.workunit.client.0.smithi191.stdout: "num_secrets": 3 2021-05-12T12:07:50.503 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.504 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 128, 2021-05-12T12:07:50.504 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_active": 128, 2021-05-12T12:07:50.504 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_unknown": 0, 2021-05-12T12:07:50.504 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osd": 3, 2021-05-12T12:07:50.504 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_sum": { 2021-05-12T12:07:50.505 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:07:50.505 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 47947, 2021-05-12T12:07:50.505 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:07:50.505 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:07:50.505 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:07:50.506 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:07:50.506 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:07:50.506 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:07:50.506 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:07:50.506 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:07:50.507 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:07:50.507 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:07:50.507 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:07:50.507 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:07:50.508 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 101, 2021-05-12T12:07:50.508 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 101, 2021-05-12T12:07:50.508 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:07:50.508 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:07:50.508 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:07:50.509 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:07:50.509 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:07:50.509 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:07:50.509 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:07:50.510 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:07:50.510 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:07:50.510 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:07:50.510 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:07:50.510 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:07:50.511 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:07:50.511 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:07:50.511 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:07:50.511 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:07:50.511 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:07:50.512 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:07:50.512 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:07:50.512 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:07:50.512 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:07:50.512 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:07:50.513 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:07:50.513 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:07:50.513 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:07:50.513 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.514 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:07:50.514 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:07:50.514 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:07:50.514 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:07:50.514 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 0, 2021-05-12T12:07:50.515 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 0, 2021-05-12T12:07:50.515 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:07:50.515 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:50.515 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:07:50.516 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:07:50.516 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:07:50.516 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.516 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 2049, 2021-05-12T12:07:50.516 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 2049, 2021-05-12T12:07:50.517 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 256, 2021-05-12T12:07:50.517 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 256, 2021-05-12T12:07:50.517 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 0 2021-05-12T12:07:50.517 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.517 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum": { 2021-05-12T12:07:50.518 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:07:50.518 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:07:50.518 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 257, 2021-05-12T12:07:50.518 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:07:50.519 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:07:50.519 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:07:50.519 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:07:50.519 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 24428, 2021-05-12T12:07:50.519 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2260, 2021-05-12T12:07:50.520 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:07:50.520 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 22141, 2021-05-12T12:07:50.520 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314548372, 2021-05-12T12:07:50.520 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:07:50.520 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:07:50.521 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322097532928, 2021-05-12T12:07:50.521 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:07:50.521 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2314240, 2021-05-12T12:07:50.521 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 501891, 2021-05-12T12:07:50.522 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:07:50.522 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:50.522 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:07:50.522 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:07:50.522 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 22673296 2021-05-12T12:07:50.523 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.523 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:07:50.523 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:07:50.523 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:07:50.523 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:07:50.524 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:07:50.524 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:07:50.524 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:07:50.524 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.525 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:07:50.525 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 92, 2021-05-12T12:07:50.525 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 92, 2021-05-12T12:07:50.525 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 92000000, 2021-05-12T12:07:50.525 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 92000000 2021-05-12T12:07:50.526 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.526 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:07:50.526 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:07:50.526 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.526 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum_by_class": { 2021-05-12T12:07:50.527 INFO:tasks.workunit.client.0.smithi191.stdout: "hdd": { 2021-05-12T12:07:50.527 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:07:50.527 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:07:50.527 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 257, 2021-05-12T12:07:50.528 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:07:50.528 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:07:50.528 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:07:50.528 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:07:50.528 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 24428, 2021-05-12T12:07:50.529 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2260, 2021-05-12T12:07:50.529 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:07:50.529 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 22141, 2021-05-12T12:07:50.529 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314548372, 2021-05-12T12:07:50.529 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:07:50.530 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:07:50.530 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322097532928, 2021-05-12T12:07:50.530 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:07:50.530 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2314240, 2021-05-12T12:07:50.531 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 501891, 2021-05-12T12:07:50.531 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:07:50.531 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:50.531 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:07:50.531 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:07:50.532 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 22673296 2021-05-12T12:07:50.532 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.532 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:07:50.532 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:07:50.532 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:07:50.533 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:07:50.533 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:07:50.533 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:07:50.533 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:07:50.534 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.534 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:07:50.534 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 92, 2021-05-12T12:07:50.534 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 92, 2021-05-12T12:07:50.534 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 92000000, 2021-05-12T12:07:50.535 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 92000000 2021-05-12T12:07:50.535 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.535 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:07:50.535 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:07:50.536 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.536 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.536 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_stats": [ 2021-05-12T12:07:50.536 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.536 INFO:tasks.workunit.client.0.smithi191.stdout: "poolid": 1, 2021-05-12T12:07:50.537 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 128, 2021-05-12T12:07:50.537 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:07:50.537 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 47947, 2021-05-12T12:07:50.537 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 100, 2021-05-12T12:07:50.537 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 0, 2021-05-12T12:07:50.538 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 300, 2021-05-12T12:07:50.538 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:07:50.538 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:07:50.538 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 100, 2021-05-12T12:07:50.539 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:07:50.539 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:07:50.539 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 100, 2021-05-12T12:07:50.539 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:07:50.539 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:07:50.540 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:07:50.540 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 101, 2021-05-12T12:07:50.540 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 101, 2021-05-12T12:07:50.540 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:07:50.540 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:07:50.541 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:07:50.541 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:07:50.541 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:07:50.541 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:07:50.541 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:07:50.542 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:07:50.542 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:07:50.542 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:07:50.542 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:07:50.543 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:07:50.543 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:07:50.543 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:07:50.543 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:07:50.543 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:07:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:07:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:07:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:07:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:07:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:07:50.545 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:07:50.545 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:07:50.545 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:07:50.545 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:07:50.546 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.546 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:07:50.546 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:07:50.546 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:07:50.547 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:07:50.547 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 1220608, 2021-05-12T12:07:50.547 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 142522, 2021-05-12T12:07:50.547 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:07:50.547 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:50.548 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:07:50.548 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:07:50.548 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:07:50.548 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.549 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 2049, 2021-05-12T12:07:50.549 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 2049, 2021-05-12T12:07:50.549 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 256, 2021-05-12T12:07:50.549 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 256, 2021-05-12T12:07:50.549 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 3 2021-05-12T12:07:50.550 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.550 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.550 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_stats": [ 2021-05-12T12:07:50.550 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.550 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:07:50.551 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 176093659156 2021-05-12T12:07:50.551 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.551 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.551 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:07:50.551 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 42949672977 2021-05-12T12:07:50.552 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.552 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.552 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:07:50.552 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 184683593747 2021-05-12T12:07:50.553 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.553 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.553 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_state": [ 2021-05-12T12:07:50.553 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.553 INFO:tasks.workunit.client.0.smithi191.stdout: "state": "active+undersized+degraded", 2021-05-12T12:07:50.554 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 100 2021-05-12T12:07:50.554 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.554 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.554 INFO:tasks.workunit.client.0.smithi191.stdout: "state": "active+undersized", 2021-05-12T12:07:50.555 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 28 2021-05-12T12:07:50.555 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.555 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.555 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_osd": [ 2021-05-12T12:07:50.555 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.556 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:07:50.556 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:07:50.556 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 128, 2021-05-12T12:07:50.556 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:07:50.556 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.557 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.557 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:07:50.557 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:07:50.557 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 0, 2021-05-12T12:07:50.557 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:07:50.558 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.558 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.558 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:07:50.558 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 128, 2021-05-12T12:07:50.559 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 128, 2021-05-12T12:07:50.559 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:07:50.559 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.559 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.560 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": [ 2021-05-12T12:07:50.560 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:07:50.560 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:07:50.560 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": {} 2021-05-12T12:07:50.560 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.561 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:07:50.561 INFO:tasks.workunit.client.0.smithi191.stdout: "servicemap": { 2021-05-12T12:07:50.561 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 6, 2021-05-12T12:07:50.561 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:07:36.236957+0000", 2021-05-12T12:07:50.561 INFO:tasks.workunit.client.0.smithi191.stdout: "services": {} 2021-05-12T12:07:50.562 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:07:50.562 INFO:tasks.workunit.client.0.smithi191.stdout: "paxos": { 2021-05-12T12:07:50.562 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:07:50.562 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 489, 2021-05-12T12:07:50.563 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pn": 0, 2021-05-12T12:07:50.563 INFO:tasks.workunit.client.0.smithi191.stdout: "accepted_pn": 0 2021-05-12T12:07:50.563 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:07:50.563 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:07:50.564 INFO:tasks.workunit.client.0.smithi191.stderr:report 2274475109 2021-05-12T12:07:50.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:07:50.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2021-05-12T12:07:50.565 INFO:tasks.workunit.client.0.smithi191.stdout:delay recovery 2021-05-12T12:07:50.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:07:50.658 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:07:50.658 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.946402+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.658 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941992+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.659 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942552+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.659 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050555+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.659 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049600+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.660 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846594+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.660 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049564+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.660 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846817+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.660 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.051030+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.661 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942290+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.661 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945780+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.661 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.847177+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.661 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942200+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833041+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947498+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833088+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942729+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.832664+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.663 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050869+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.663 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.837911+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.663 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945483+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.663 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833729+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941846+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833578+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947161+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833814+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942362+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833464+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050707+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840401+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945174+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839289+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.666 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.666 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942000+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.666 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942817+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.666 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840197+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.667 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941588+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.667 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838547+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.667 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050396+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.667 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843504+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846242+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841485+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846461+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943327+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941935+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.669 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842661+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.669 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846589+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.669 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842451+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.669 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832706+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.670 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838298+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.670 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837624+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.670 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945419+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.670 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842097+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844805+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839883+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844828+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942564+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941545+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.672 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841976+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.672 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845732+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.672 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840522+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.672 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050125+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844172+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845879+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842251+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846106+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.944548+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.674 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.847030+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.674 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843740+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.674 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845368+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.674 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845354+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945126+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845891+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844452+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.844480+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844120+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.676 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.676 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845162+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.676 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845398+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.676 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945706+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844119+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842974+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844093+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844900+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:06:38.944832+0000 48'101 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.678 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.678 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844564+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.678 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839359+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.678 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846643+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.679 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941056+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.679 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845734+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.679 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838171+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.679 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845534+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839523+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945992+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842066+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842614+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842320+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844527+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943028+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842817+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840183+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842977+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.682 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842824+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.682 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943625+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.682 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838009+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.682 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841875+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.683 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839833+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.683 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843815+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.683 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.684 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839918+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.684 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838521+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.684 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841628+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.684 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839901+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942281+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833094+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839602+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833452+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841714+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833276+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833821+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833336+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839029+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837516+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837177+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.715297+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837775+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832674+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:50.688 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:07:50.688 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:07:50.689 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:07:50.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:07:50.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2021-05-12T12:07:50.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:07:50.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:07:50.691 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:07:50.691 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:07:50.691 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:07:50.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.0.asok 2021-05-12T12:07:50.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:07:50.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T12:07:50.766 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:07:50.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:07:50.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2021-05-12T12:07:50.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:07:50.777 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:07:50.777 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:07:50.777 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:07:50.777 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:07:50.777 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.2.asok 2021-05-12T12:07:50.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:07:50.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T12:07:50.872 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:07:50.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2021-05-12T12:07:50.881 INFO:tasks.workunit.client.0.smithi191.stdout:revive divergent 1 2021-05-12T12:07:50.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:07:51.226 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:07:51.226 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.946402+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.226 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941992+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.227 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942552+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.227 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050555+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.227 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049600+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.227 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846594+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049564+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846817+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.051030+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942290+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945780+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.229 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.847177+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.229 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942200+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.229 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833041+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.229 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947498+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.229 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833088+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.230 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942729+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.230 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.832664+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.230 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050869+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.230 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.837911+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945483+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833729+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941846+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833578+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947161+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.232 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833814+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.232 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942362+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.232 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833464+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.232 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050707+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840401+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945174+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839289+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942000+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.234 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942817+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.234 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840197+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.234 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941588+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.234 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838547+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.234 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050396+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843504+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846242+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841485+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846461+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943327+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941935+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842661+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846589+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842451+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832706+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838298+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837624+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945419+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842097+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844805+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839883+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844828+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942564+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941545+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.239 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841976+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.239 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845732+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.239 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840522+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.239 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050125+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.240 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844172+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.240 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845879+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.240 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842251+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.240 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846106+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.240 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.944548+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.241 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.847030+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.241 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843740+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.241 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845368+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.241 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845354+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945126+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845891+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844452+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.844480+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844120+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845162+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845398+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945706+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844119+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842974+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844093+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844900+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:06:38.944832+0000 48'101 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844564+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839359+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846643+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941056+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845734+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.246 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838171+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.246 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845534+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.246 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839523+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.246 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945992+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842066+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842614+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842320+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844527+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943028+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842817+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840183+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842977+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842824+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943625+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.249 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838009+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.249 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841875+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.249 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839833+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.249 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843815+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839918+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838521+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841628+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839901+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942281+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833094+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839602+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833452+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841714+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.252 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833276+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.252 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833821+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.252 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833336+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.252 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839029+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.253 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837516+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.253 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837177+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.253 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.715297+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.253 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837775+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.253 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832674+0000 31'100 48:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:51.253 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:07:51.254 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:07:51.254 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:07:51.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2021-05-12T12:07:51.983 INFO:tasks.workunit.client.0.smithi191.stderr:noup is set 2021-05-12T12:07:51.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2021-05-12T12:07:51.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:07:51.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:07:51.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:07:51.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:07:51.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:07:51.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:07:51.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:07:51.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:07:51.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:07:51.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:07:51.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:07:51.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:07:51.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:07:52.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:07:52.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:07:52.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:07:52.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:07:52.000 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:07:52.001 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:07:52.001 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:07:52.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:07:52.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:07:52.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:07:52.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:07:52.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:07:52.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:07:52.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:07:52.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:07:52.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:07:52.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:07:52.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:07:52.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:07:52.004 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:07:52.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:07:52.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:07:52.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:07:52.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:07:52.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:07:52.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:07:52.023 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:07:52.021+0000 7f165f0270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:07:52.044 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:07:52.041+0000 7f165f0270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:07:52.045 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:07:52.043+0000 7f165f0270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:07:52.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T12:07:52.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2021-05-12T12:07:52.564 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:07:52.560+0000 7f165f0270c0 -1 Falling back to public interface 2021-05-12T12:07:55.449 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:07:55.446+0000 7f165f0270c0 -1 osd.1 34 log_to_monitors {default=true} 2021-05-12T12:07:57.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2021-05-12T12:07:57.399 INFO:tasks.workunit.client.0.smithi191.stdout:delay recovery divergent 2021-05-12T12:07:57.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:07:57.753 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:07:57.754 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.946402+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.754 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941992+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.754 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942552+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.754 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050555+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.755 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049600+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.755 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846594+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.755 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049564+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.755 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846817+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.755 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.051030+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.756 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942290+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.756 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945780+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.756 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.847177+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.756 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942200+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.757 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833041+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.757 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947498+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.757 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833088+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.757 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942729+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.757 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.832664+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.758 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050869+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.758 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.837911+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.758 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945483+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.758 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833729+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.759 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941846+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.759 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833578+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.759 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947161+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.759 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833814+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.759 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942362+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.760 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833464+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.760 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050707+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.760 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840401+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.760 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945174+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.760 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839289+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.761 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.761 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942000+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.761 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942817+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.761 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840197+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.761 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941588+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.762 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838547+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.762 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050396+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.762 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843504+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.762 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846242+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.762 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841485+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.763 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846461+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.763 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943327+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.763 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941935+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.763 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842661+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.764 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846589+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.764 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842451+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.764 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832706+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.764 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838298+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.764 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837624+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.765 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945419+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.765 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842097+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.765 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844805+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.765 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839883+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.765 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844828+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.766 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942564+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.766 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941545+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.766 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841976+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.766 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845732+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.767 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840522+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.767 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050125+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.767 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844172+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.767 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845879+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.767 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842251+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.768 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846106+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.768 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.944548+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.768 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.847030+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.768 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843740+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.768 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845368+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.769 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845354+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.769 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945126+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.769 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845891+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.769 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844452+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.770 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.844480+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.770 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844120+0000 31'100 48:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.770 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.770 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845162+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.770 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845398+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.771 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945706+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.771 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844119+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.771 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842974+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.771 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844093+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.771 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844900+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.772 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:06:38.944832+0000 48'101 48:129 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.772 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.772 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844564+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.772 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839359+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.772 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846643+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.773 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941056+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.773 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845734+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.773 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838171+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.773 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845534+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.774 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839523+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.774 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945992+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.774 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842066+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.774 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842614+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.774 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842320+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.775 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844527+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.775 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943028+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.781 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842817+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.782 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840183+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.782 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842977+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.782 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842824+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.782 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943625+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.783 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838009+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.783 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841875+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.783 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839833+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.783 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843815+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.783 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.784 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839918+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.784 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838521+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.784 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841628+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.784 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839901+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.785 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942281+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.785 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833094+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.785 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839602+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.785 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833452+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.785 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841714+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.786 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833276+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.786 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833821+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.786 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833336+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.786 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839029+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.786 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837516+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.787 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837177+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.787 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.715297+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.787 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837775+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.787 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832674+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:07:57.788 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:07:57.788 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:07:57.789 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:07:57.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2021-05-12T12:07:57.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:07:57.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:07:57.790 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:07:57.790 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:07:57.791 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:07:57.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.1.asok 2021-05-12T12:07:57.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:07:57.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T12:07:57.859 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:07:57.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2021-05-12T12:07:58.533 INFO:tasks.workunit.client.0.smithi191.stderr:noup is unset 2021-05-12T12:07:58.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2021-05-12T12:07:58.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:07:58.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:07:58.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:07:58.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:07:58.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:07:58.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:07:58.544 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:07:58.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:07:58.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:07:59.010 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 41 up_thru 47 down_at 35 last_clean_interval [5,31) [v2:127.0.0.1:6802/67634,v1:127.0.0.1:6803/67634] [v2:127.0.0.1:6804/67634,v1:127.0.0.1:6805/67634] exists,up efb617d0-a38f-4017-94d3-fc68ac2ed9be 2021-05-12T12:07:59.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:07:59.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:07:59.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:07:59.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2021-05-12T12:07:59.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:07:59.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:07:59.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:07:59.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:07:59.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:07:59.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:07:59.014 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:07:59.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:07:59.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:07:59.427 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 36 last_clean_interval [10,34) [v2:127.0.0.1:6818/70933,v1:127.0.0.1:6819/70933] [v2:127.0.0.1:6820/70933,v1:127.0.0.1:6821/70933] exists,up 5fe20a5a-c3fb-42c5-aadd-2c4671a13114 2021-05-12T12:07:59.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:07:59.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:07:59.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:07:59.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2021-05-12T12:07:59.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:07:59.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:07:59.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:07:59.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:07:59.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:07:59.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:07:59.431 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:07:59.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:07:59.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:07:59.821 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 43 up_thru 0 down_at 37 last_clean_interval [15,31) [v2:127.0.0.1:6810/68217,v1:127.0.0.1:6811/68217] [v2:127.0.0.1:6812/68217,v1:127.0.0.1:6813/68217] exists,up 16b8b89a-4aa1-4296-88f5-d60bf3eabab1 2021-05-12T12:07:59.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:07:59.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:07:59.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:07:59.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:00.225 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:00.226 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.946402+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.226 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941992+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.226 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942552+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.227 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050555+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.227 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049600+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.227 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846594+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.227 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049564+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.227 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846817+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.051030+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942290+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945780+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.847177+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.228 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942200+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.229 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833041+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.229 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947498+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.229 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833088+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.229 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942729+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.230 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.832664+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.230 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050869+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.230 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.837911+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.230 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945483+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.230 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833729+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941846+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833578+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947161+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833814+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.231 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942362+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.232 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833464+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.232 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050707+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.232 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840401+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.232 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945174+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.232 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839289+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942000+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942817+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840197+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.233 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941588+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.234 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838547+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.234 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050396+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.234 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843504+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.234 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846242+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841485+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846461+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943327+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941935+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.235 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842661+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846589+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842451+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832706+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838298+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837624+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945419+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842097+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844805+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839883+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.237 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844828+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942564+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941545+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841976+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845732+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.238 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840522+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.239 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050125+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.239 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844172+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.239 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845879+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.239 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842251+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.239 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846106+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.240 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.944548+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.240 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.847030+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.240 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843740+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.240 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845368+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.241 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845354+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.241 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945126+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.241 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845891+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.241 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844452+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.241 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.844480+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844120+0000 31'100 48:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845162+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845398+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.242 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945706+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844119+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842974+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844093+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844900+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.243 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:06:38.944832+0000 48'101 48:129 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844564+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839359+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846643+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.244 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941056+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845734+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838171+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845534+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839523+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.245 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945992+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.246 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842066+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.246 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842614+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.246 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842320+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.246 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844527+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943028+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842817+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840183+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842977+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.247 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842824+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943625+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838009+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841875+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839833+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843815+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.248 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.249 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839918+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.249 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838521+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.249 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841628+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.249 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839901+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.249 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942281+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833094+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839602+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833452+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841714+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.250 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833276+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833821+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833336+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839029+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837516+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.251 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837177+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.252 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.715297+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.252 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837775+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.252 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832674+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.252 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:08:00.252 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:00.253 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:08:00.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2021-05-12T12:08:00.254 INFO:tasks.workunit.client.0.smithi191.stdout:wait for peering 2021-05-12T12:08:00.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:00.590 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:00.590 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.946402+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.591 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941992+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.591 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942552+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.591 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050555+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.591 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049600+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.591 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846594+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.592 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.049564+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.592 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.846817+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.592 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.051030+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.592 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942290+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.592 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945780+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.593 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.847177+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.593 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942200+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.593 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833041+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.593 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947498+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.594 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833088+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.594 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942729+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.594 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.832664+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.594 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:39.050869+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.594 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.837911+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.595 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.945483+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.595 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833729+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.595 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.941846+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.595 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833578+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.596 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.947161+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.596 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.833814+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.596 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.942362+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.596 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833464+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.596 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050707+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.597 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840401+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.597 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945174+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.597 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839289+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.597 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.598 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942000+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.598 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942817+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.598 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840197+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.598 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941588+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.599 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838547+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.599 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050396+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.599 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843504+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.599 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846242+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.599 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841485+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.600 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846461+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.600 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943327+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.600 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941935+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.600 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842661+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.601 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846589+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.602 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842451+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.602 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832706+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.602 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838298+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.602 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837624+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.603 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945419+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.603 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842097+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.603 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844805+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.603 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839883+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.603 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844828+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.604 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942564+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.604 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941545+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.604 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841976+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.604 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845732+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.605 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840522+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.605 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:39.050125+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.605 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844172+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.606 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845879+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.606 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842251+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.606 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846106+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.606 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.944548+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.606 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.847030+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.607 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843740+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.607 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845368+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.607 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845354+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.607 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945126+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.607 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845891+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.608 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844452+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.608 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:06:38.844480+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.608 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844120+0000 31'100 48:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.608 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.609 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845162+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.609 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845398+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.609 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945706+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.609 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844119+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.609 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842974+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.610 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844093+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.610 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844900+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.610 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:06:38.944832+0000 48'101 48:129 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.610 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843745+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.610 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844564+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.611 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839359+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.611 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.846643+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.611 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941056+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.611 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845734+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.611 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838171+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.612 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.845534+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.612 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839523+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.612 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.945992+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.612 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842066+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.612 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842614+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.613 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842320+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.613 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.844527+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.613 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943028+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.613 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842817+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.614 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.840183+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.614 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842977+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.614 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.842824+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.614 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.943625+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.614 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838009+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.615 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841875+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.615 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839833+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.615 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.843815+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.615 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.941539+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.616 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839918+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.616 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.838521+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.616 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841628+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.616 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839901+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.616 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.942281+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.617 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833094+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.617 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839602+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.617 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833452+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.617 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.841714+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.617 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833276+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.618 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833821+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.618 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.833336+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.618 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.839029+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.618 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837516+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.618 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837177+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.619 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.715297+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.619 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.837775+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.619 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:06:38.832674+0000 31'100 48:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:00.619 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:08:00.620 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:00.620 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:08:00.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2021-05-12T12:08:07.118 INFO:tasks.workunit.client.0.smithi191.stdout:killing divergent 1 2021-05-12T12:08:07.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2021-05-12T12:08:07.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:07.481 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:07.481 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928873+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.482 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928809+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.482 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.757139+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.482 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.758549+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.482 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.935184+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.483 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927694+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.483 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929654+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.483 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755961+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.483 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927251+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.484 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.934161+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.484 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928612+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.484 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928217+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.484 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756794+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.484 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.657506+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.485 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.934899+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.485 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.651348+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.485 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929428+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.485 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.656581+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.486 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.926915+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.486 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.926680+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.486 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.928252+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.486 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.652811+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.486 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 1 1 0 0 377 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:07:59.759295+0000 31'100 54:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.487 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.661663+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.487 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.934684+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.487 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.652283+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.487 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929119+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.488 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748295+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.488 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.759416+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.488 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928505+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.488 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927982+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.488 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748895+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.489 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756186+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.489 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.664508+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.489 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.934421+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.489 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.664291+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.489 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928538+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.490 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.749332+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.490 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.758027+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.490 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929380+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.490 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927248+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.491 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753157+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.491 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755568+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.491 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.744898+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.491 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.933904+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.491 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.745284+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.492 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927973+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.492 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.752026+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.492 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.652608+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.492 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927374+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.492 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748833+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.493 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756132+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.493 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 0 0 0 378 0 0 15 15 active+clean+wait 2021-05-12T12:07:59.928926+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.493 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.749911+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.493 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.752024+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.494 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753894+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.494 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.744285+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.494 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.933713+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.494 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 1 1 0 0 378 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:07:59.929088+0000 31'100 54:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.494 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927716+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.495 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750359+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.495 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.757395+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.495 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.930150+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.495 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.751542+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.495 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753999+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.496 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755323+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.496 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.745853+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.496 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.933438+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.496 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 1 1 0 0 378 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:07:59.929513+0000 31'100 54:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.496 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.927425+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.497 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.754314+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.497 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755861+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.497 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.932639+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.497 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.749015+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.497 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753682+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.498 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756417+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.498 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755894+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.498 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750597+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.498 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.932407+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.498 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756363+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.499 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753417+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.499 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755151+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.499 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 0 0 0 378 0 0 14 14 active+clean+wait 2021-05-12T12:07:59.746866+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.499 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.932126+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.500 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 1 1 0 0 25 0 0 14 14 active+recovering+degraded+wait 2021-05-12T12:07:59.928963+0000 48'101 54:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.500 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.752983+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.500 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927139+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.500 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.657917+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.500 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.933163+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.501 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.662431+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.501 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.754627+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.501 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748336+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.501 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.751190+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.501 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927616+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.502 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756724+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.502 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750606+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.502 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.754502+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.502 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.665074+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.503 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.931807+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.503 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.744682+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.503 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.752594+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.503 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.752908+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.503 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.745893+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.504 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929131+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.504 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.745163+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.504 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.749063+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.504 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.753385+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.504 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.664064+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.505 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929734+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.505 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.662808+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.505 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750081+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.505 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748687+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.505 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.664707+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.506 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928002+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.506 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.664909+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.506 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.744773+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.506 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750794+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.506 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.652079+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.507 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928675+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.507 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.660808+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.507 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748613+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.507 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.609689+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.508 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.657188+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.508 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 0 0 0 377 0 0 11 11 active+clean+wait 2021-05-12T12:07:59.744107+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.508 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.661928+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.508 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.610775+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.508 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.662093+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.509 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 0 0 0 377 0 0 12 12 active+clean+wait 2021-05-12T12:07:59.478007+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:07.509 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:08:07.509 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:07.510 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:08:07.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2021-05-12T12:08:07.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:08:07.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:08:07.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:08:07.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:08:07.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:08:07.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:08:07.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.34183.out 2021-05-12T12:08:07.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.34183.out 2021-05-12T12:08:07.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:08:07.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:08:07.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:08:07.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:08:07.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:08:07.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.34183.out 2021-05-12T12:08:08.560 INFO:tasks.workunit.client.0.smithi191.stderr:Exporting 1.0 info 1.0( v 31'100 (31'84,31'100] local-lis/les=53/54 n=0 ec=23/23 lis/c=53/53 les/c/f=54/54/0 sis=53 pruub=7.264920235s) 2021-05-12T12:08:08.564 INFO:tasks.workunit.client.0.smithi191.stderr:Export successful 2021-05-12T12:08:08.565 INFO:tasks.workunit.client.0.smithi191.stdout: marking collection for removal 2021-05-12T12:08:08.565 INFO:tasks.workunit.client.0.smithi191.stdout:setting '_remove' omap key 2021-05-12T12:08:08.565 INFO:tasks.workunit.client.0.smithi191.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-05-12T12:08:08.751 INFO:tasks.workunit.client.0.smithi191.stderr:Remove successful 2021-05-12T12:08:09.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.34183.out 2021-05-12T12:08:09.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:08:09.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:08:09.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:08:09.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:08:09.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:08:09.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.34183.out 2021-05-12T12:08:09.783 INFO:tasks.workunit.client.0.smithi191.stdout:get_pg_num_history pg_num_history pg_num_history(e54 pg_nums {1={23=1,33=128}} deleted_pools ) 2021-05-12T12:08:09.784 INFO:tasks.workunit.client.0.smithi191.stdout:Importing pgid 1.0 2021-05-12T12:08:09.784 INFO:tasks.workunit.client.0.smithi191.stdout:write_pg epoch 54 info 1.0( v 31'100 (31'84,31'100] local-lis/les=53/54 n=0 ec=23/23 lis/c=53/53 les/c/f=54/54/0 sis=53 pruub=7.264920235s) 2021-05-12T12:08:09.853 INFO:tasks.workunit.client.0.smithi191.stdout:Import successful 2021-05-12T12:08:10.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2021-05-12T12:08:10.300 INFO:tasks.workunit.client.0.smithi191.stdout:reviving divergent 1 2021-05-12T12:08:10.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:10.660 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:10.661 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928873+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.661 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928809+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.661 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.757139+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.661 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.758549+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.935184+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927694+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929654+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755961+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.662 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927251+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.663 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.934161+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.663 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928612+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.663 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928217+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.663 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756794+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.657506+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.934899+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.651348+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929428+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.664 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.656581+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.926915+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.926680+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.928252+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.652811+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.665 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 1 1 0 0 377 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:07:59.759295+0000 31'100 54:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.666 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.661663+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.666 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.934684+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.666 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.652283+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.666 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929119+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.667 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748295+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.667 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.759416+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.667 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928505+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.667 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927982+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.667 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748895+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756186+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.664508+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.934421+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.664291+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.668 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928538+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.669 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.749332+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.669 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.758027+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.669 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929380+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.669 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927248+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.669 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753157+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.670 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755568+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.670 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.744898+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.670 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.933904+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.670 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.745284+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927973+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.752026+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.652608+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927374+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748833+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.672 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756132+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.672 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 0 0 0 378 0 0 15 15 active+clean+wait 2021-05-12T12:07:59.928926+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.672 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.749911+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.672 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.752024+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753894+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.744285+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.933713+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 1 1 0 0 378 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:07:59.929088+0000 31'100 54:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.673 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927716+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.674 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750359+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.674 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.757395+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.674 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.930150+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.674 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.751542+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.674 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753999+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755323+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.745853+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.933438+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 1 1 0 0 378 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:07:59.929513+0000 31'100 54:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.675 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.927425+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.676 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.754314+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.676 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755861+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.676 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.932639+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.676 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.749015+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.676 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753682+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756417+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755894+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750597+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.932407+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.677 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756363+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.678 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.753417+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.678 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.755151+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.678 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 0 0 0 378 0 0 14 14 active+clean+wait 2021-05-12T12:07:59.746866+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.678 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.932126+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.678 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 1 1 0 0 25 0 0 14 14 active+recovering+degraded+wait 2021-05-12T12:07:59.928963+0000 48'101 54:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.679 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.752983+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.679 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927139+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.679 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.657917+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.679 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.933163+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.679 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.662431+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.754627+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748336+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.751190+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.927616+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.680 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.756724+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750606+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.754502+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.665074+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.931807+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.681 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.744682+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.682 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.752594+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.682 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.752908+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.682 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.745893+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.682 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929131+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.682 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.745163+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.683 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.749063+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.683 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.753385+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.683 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.664064+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.683 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.929734+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.683 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.662808+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.684 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750081+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.684 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748687+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.684 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.664707+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.684 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928002+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.684 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.664909+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.744773+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.750794+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.652079+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.928675+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.685 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.660808+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.748613+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.609689+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:07:59.657188+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 0 0 0 377 0 0 11 11 active+clean+wait 2021-05-12T12:07:59.744107+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.661928+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.610775+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:07:59.662093+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.687 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 0 0 0 377 0 0 12 12 active+clean+wait 2021-05-12T12:07:59.478007+0000 31'100 54:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:10.687 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:08:10.687 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:10.691 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:08:10.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2021-05-12T12:08:10.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:08:10.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:08:10.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:08:10.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:08:10.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:08:10.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:08:10.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:08:10.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:08:10.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:08:10.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:08:10.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:08:10.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:08:10.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:08:10.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:08:10.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:08:10.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:10.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:10.695 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:10.695 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:10.695 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:10.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:10.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:10.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:08:10.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:08:10.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:08:10.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:08:10.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:08:10.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:10.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:08:10.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:08:10.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:08:10.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:08:10.698 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:08:10.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:10.699 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:08:10.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:08:10.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:08:10.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:08:10.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:08:10.702 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:10.699+0000 7fa7a47ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:10.704 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:10.701+0000 7fa7a47ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:10.706 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:10.703+0000 7fa7a47ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:11.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:08:11.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:08:11.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:08:11.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:08:11.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:08:11.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:11.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:08:11.043 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:08:11.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:11.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:08:11.221 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:11.219+0000 7fa7a47ff0c0 -1 Falling back to public interface 2021-05-12T12:08:11.457 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 36 last_clean_interval [10,34) [v2:127.0.0.1:6818/70933,v1:127.0.0.1:6819/70933] [v2:127.0.0.1:6820/70933,v1:127.0.0.1:6821/70933] exists,up 5fe20a5a-c3fb-42c5-aadd-2c4671a13114 2021-05-12T12:08:11.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:08:11.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:08:11.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:08:11.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2021-05-12T12:08:11.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:08:11.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:08:11.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:08:11.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:08:11.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:11.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:08:11.461 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:08:11.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:11.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:08:11.824 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 36 last_clean_interval [10,34) [v2:127.0.0.1:6818/70933,v1:127.0.0.1:6819/70933] [v2:127.0.0.1:6820/70933,v1:127.0.0.1:6821/70933] exists,up 5fe20a5a-c3fb-42c5-aadd-2c4671a13114 2021-05-12T12:08:11.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:08:11.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:08:11.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:08:11.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2021-05-12T12:08:13.625 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:13.622+0000 7fa7a47ff0c0 -1 osd.1 54 log_to_monitors {default=true} 2021-05-12T12:08:31.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2021-05-12T12:08:31.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:08:31.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:08:31.828 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:08:31.828 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:31.829 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:31.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.1.asok 2021-05-12T12:08:31.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:08:31.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.1.asok dump_ops_in_flight 2021-05-12T12:08:31.928 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:08:31.929 INFO:tasks.workunit.client.0.smithi191.stdout: "ops": [], 2021-05-12T12:08:31.929 INFO:tasks.workunit.client.0.smithi191.stdout: "num_ops": 0 2021-05-12T12:08:31.929 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:08:31.937 INFO:tasks.workunit.client.0.smithi191.stdout:allowing recovery 2021-05-12T12:08:31.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2021-05-12T12:08:31.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:32.282 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:32.283 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.378458+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.283 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.348012+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.283 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.097239+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.284 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.093844+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.284 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.586173+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.284 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098659+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.284 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098203+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.284 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.268342+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.285 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.268880+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.285 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.097410+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.285 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.378218+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.285 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.262948+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.286 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.585326+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.286 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.265536+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.286 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.378301+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.286 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.260667+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.287 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.378790+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.287 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266378+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.287 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.268702+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.287 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266049+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.287 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.099218+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.288 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095052+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.288 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.348939+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.288 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.265803+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.288 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.585651+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.288 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096580+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.289 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.263369+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.289 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.094755+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.289 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.587075+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.289 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:26.743404+0000 31'100 62:130 [2,0,1] 2 [2,0,1] 2 31'100 2021-05-12T12:08:26.743337+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.290 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098979+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.290 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096167+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.290 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.348750+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.290 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.003898+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.291 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.097727+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.291 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266775+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.291 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098007+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.291 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.261811+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.291 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.093319+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.292 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.264376+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.292 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098343+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.292 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.270103+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.292 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096921+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.292 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.004538+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.293 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.271755+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.293 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.267222+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.293 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.272789+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.293 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095616+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.293 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.261149+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.294 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.094507+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.294 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.094976+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.294 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.267256+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.294 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 0 0 0 378 0 0 15 15 active+clean 2021-05-12T12:08:15.508021+0000 31'100 61:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.294 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.270428+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.295 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.096795+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.295 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.511746+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.295 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.508686+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.295 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097095+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.295 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097338+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.296 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.262267+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.296 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.509458+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.296 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.268438+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.296 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.095571+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.296 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.507583+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.297 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.448228+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.297 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.348555+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.297 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.510383+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.297 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.096804+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.297 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097983+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.298 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097792+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.298 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.348287+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.298 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.514681+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.298 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 0 0 0 378 0 0 10 10 active+clean 2021-05-12T12:08:26.264924+0000 31'100 62:160 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.299 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.098138+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.299 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.511457+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.299 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.272401+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.299 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 2 0 0 0 0 861 0 0 17 17 active+clean 2021-05-12T12:08:15.507024+0000 54'101 61:151 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.299 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.270803+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.300 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.264708+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.300 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.263225+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.300 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.267932+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.300 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.448767+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.301 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 0 0 0 378 0 0 14 14 active+clean 2021-05-12T12:08:15.448740+0000 31'100 61:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.301 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.267160+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.301 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 0 0 0 25 0 0 14 14 active+clean 2021-05-12T12:08:26.263024+0000 48'101 62:162 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.301 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.348108+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.301 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097406+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.302 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.265942+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.302 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.265323+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.302 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.262839+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.302 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.096670+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.303 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.266935+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.303 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.270611+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.303 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.516964+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.303 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.268151+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.303 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.272722+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.304 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.262059+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.304 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.097618+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.304 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096024+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.304 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.004138+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.304 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095890+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.305 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.269201+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.305 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.270232+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.305 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:26.503487+0000 31'100 62:149 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:26.503429+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.305 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266935+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.305 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:28.711788+0000 31'100 62:154 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:28.711720+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.306 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 0 0 0 377 0 0 10 10 active+clean 2021-05-12T12:08:15.447593+0000 31'100 61:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.306 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096925+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.306 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095289+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.306 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.003550+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.307 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095256+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.307 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095883+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.307 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.265635+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.307 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.516826+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.307 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.508296+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.308 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.261407+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.308 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096392+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.308 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.265740+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.308 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.264171+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.308 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.448214+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.309 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266582+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.309 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.093933+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.309 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096751+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.309 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.093204+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.309 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-05-12T12:08:29.468800+0000 31'100 62:149 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:29.468737+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.310 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.099050+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.310 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:25.806920+0000 31'100 62:154 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:25.806836+0000 31'100 2021-05-12T12:08:25.806836+0000 0 2021-05-12T12:08:32.310 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 0 0 0 377 0 0 12 12 active+clean 2021-05-12T12:08:26.264985+0000 31'100 62:160 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:32.310 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:08:32.310 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:32.311 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:08:32.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-05-12T12:08:32.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T12:08:32.417 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:08:32.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-05-12T12:08:32.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T12:08:32.551 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:08:32.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-05-12T12:08:32.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T12:08:32.683 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:08:32.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2021-05-12T12:08:32.693 INFO:tasks.workunit.client.0.smithi191.stdout:reading divergent objects 2021-05-12T12:08:32.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:33.037 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:33.037 INFO:tasks.workunit.client.0.smithi191.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.378458+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.038 INFO:tasks.workunit.client.0.smithi191.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.348012+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.038 INFO:tasks.workunit.client.0.smithi191.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.097239+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.038 INFO:tasks.workunit.client.0.smithi191.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.093844+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.038 INFO:tasks.workunit.client.0.smithi191.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.586173+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.039 INFO:tasks.workunit.client.0.smithi191.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098659+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.039 INFO:tasks.workunit.client.0.smithi191.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098203+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.039 INFO:tasks.workunit.client.0.smithi191.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.268342+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.039 INFO:tasks.workunit.client.0.smithi191.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.268880+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.039 INFO:tasks.workunit.client.0.smithi191.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.097410+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.040 INFO:tasks.workunit.client.0.smithi191.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.378218+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.040 INFO:tasks.workunit.client.0.smithi191.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.262948+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.040 INFO:tasks.workunit.client.0.smithi191.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.585326+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.040 INFO:tasks.workunit.client.0.smithi191.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.265536+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.041 INFO:tasks.workunit.client.0.smithi191.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.378301+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.041 INFO:tasks.workunit.client.0.smithi191.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.260667+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.041 INFO:tasks.workunit.client.0.smithi191.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.378790+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.041 INFO:tasks.workunit.client.0.smithi191.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266378+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.041 INFO:tasks.workunit.client.0.smithi191.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.268702+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.042 INFO:tasks.workunit.client.0.smithi191.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266049+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.042 INFO:tasks.workunit.client.0.smithi191.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.099218+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.042 INFO:tasks.workunit.client.0.smithi191.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095052+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.042 INFO:tasks.workunit.client.0.smithi191.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.348939+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.042 INFO:tasks.workunit.client.0.smithi191.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.265803+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.043 INFO:tasks.workunit.client.0.smithi191.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.585651+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.043 INFO:tasks.workunit.client.0.smithi191.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096580+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.043 INFO:tasks.workunit.client.0.smithi191.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.263369+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.043 INFO:tasks.workunit.client.0.smithi191.stdout:1.64 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.094755+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.043 INFO:tasks.workunit.client.0.smithi191.stdout:1.63 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.587075+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.044 INFO:tasks.workunit.client.0.smithi191.stdout:1.62 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:26.743404+0000 31'100 62:130 [2,0,1] 2 [2,0,1] 2 31'100 2021-05-12T12:08:26.743337+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.044 INFO:tasks.workunit.client.0.smithi191.stdout:1.61 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098979+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.044 INFO:tasks.workunit.client.0.smithi191.stdout:1.60 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096167+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.044 INFO:tasks.workunit.client.0.smithi191.stdout:1.5f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.348750+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.045 INFO:tasks.workunit.client.0.smithi191.stdout:1.5e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.003898+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.045 INFO:tasks.workunit.client.0.smithi191.stdout:1.5d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.097727+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.045 INFO:tasks.workunit.client.0.smithi191.stdout:1.5c 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266775+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.045 INFO:tasks.workunit.client.0.smithi191.stdout:1.5b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098007+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.045 INFO:tasks.workunit.client.0.smithi191.stdout:1.5a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.261811+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.046 INFO:tasks.workunit.client.0.smithi191.stdout:1.59 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.093319+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.046 INFO:tasks.workunit.client.0.smithi191.stdout:1.58 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.264376+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.046 INFO:tasks.workunit.client.0.smithi191.stdout:1.57 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.098343+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.046 INFO:tasks.workunit.client.0.smithi191.stdout:1.56 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.270103+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.046 INFO:tasks.workunit.client.0.smithi191.stdout:1.55 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096921+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.047 INFO:tasks.workunit.client.0.smithi191.stdout:1.54 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.004538+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.047 INFO:tasks.workunit.client.0.smithi191.stdout:1.53 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.271755+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.047 INFO:tasks.workunit.client.0.smithi191.stdout:1.52 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.267222+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.047 INFO:tasks.workunit.client.0.smithi191.stdout:1.51 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.272789+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.048 INFO:tasks.workunit.client.0.smithi191.stdout:1.50 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095616+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.048 INFO:tasks.workunit.client.0.smithi191.stdout:1.4f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.261149+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.048 INFO:tasks.workunit.client.0.smithi191.stdout:1.4e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.094507+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.048 INFO:tasks.workunit.client.0.smithi191.stdout:1.23 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.094976+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.048 INFO:tasks.workunit.client.0.smithi191.stdout:1.22 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.267256+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.049 INFO:tasks.workunit.client.0.smithi191.stdout:1.21 1 0 0 0 0 378 0 0 15 15 active+clean 2021-05-12T12:08:15.508021+0000 31'100 61:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.049 INFO:tasks.workunit.client.0.smithi191.stdout:1.20 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.270428+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.049 INFO:tasks.workunit.client.0.smithi191.stdout:1.1f 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.096795+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.049 INFO:tasks.workunit.client.0.smithi191.stdout:1.1e 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.511746+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.050 INFO:tasks.workunit.client.0.smithi191.stdout:1.1d 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.508686+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.050 INFO:tasks.workunit.client.0.smithi191.stdout:1.1c 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097095+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.050 INFO:tasks.workunit.client.0.smithi191.stdout:1.1b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097338+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.050 INFO:tasks.workunit.client.0.smithi191.stdout:1.1a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.262267+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.050 INFO:tasks.workunit.client.0.smithi191.stdout:1.19 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.509458+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.051 INFO:tasks.workunit.client.0.smithi191.stdout:1.18 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.268438+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.051 INFO:tasks.workunit.client.0.smithi191.stdout:1.17 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.095571+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.051 INFO:tasks.workunit.client.0.smithi191.stdout:1.16 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.507583+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.051 INFO:tasks.workunit.client.0.smithi191.stdout:1.15 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.448228+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.051 INFO:tasks.workunit.client.0.smithi191.stdout:1.14 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.348555+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.052 INFO:tasks.workunit.client.0.smithi191.stdout:1.13 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.510383+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.052 INFO:tasks.workunit.client.0.smithi191.stdout:1.12 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.096804+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.052 INFO:tasks.workunit.client.0.smithi191.stdout:1.11 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097983+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.052 INFO:tasks.workunit.client.0.smithi191.stdout:1.10 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097792+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.053 INFO:tasks.workunit.client.0.smithi191.stdout:1.5 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.348287+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.053 INFO:tasks.workunit.client.0.smithi191.stdout:1.4 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.514681+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.053 INFO:tasks.workunit.client.0.smithi191.stdout:1.3 1 0 0 0 0 378 0 0 10 10 active+clean 2021-05-12T12:08:26.264924+0000 31'100 62:160 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.053 INFO:tasks.workunit.client.0.smithi191.stdout:1.2 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.098138+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.053 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.511457+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.054 INFO:tasks.workunit.client.0.smithi191.stdout:1.1 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.272401+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.054 INFO:tasks.workunit.client.0.smithi191.stdout:1.6 2 0 0 0 0 861 0 0 17 17 active+clean 2021-05-12T12:08:15.507024+0000 54'101 61:151 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.054 INFO:tasks.workunit.client.0.smithi191.stdout:1.7 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.270803+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.054 INFO:tasks.workunit.client.0.smithi191.stdout:1.8 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.264708+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.054 INFO:tasks.workunit.client.0.smithi191.stdout:1.9 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.263225+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.055 INFO:tasks.workunit.client.0.smithi191.stdout:1.a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.267932+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.055 INFO:tasks.workunit.client.0.smithi191.stdout:1.b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.448767+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.055 INFO:tasks.workunit.client.0.smithi191.stdout:1.c 1 0 0 0 0 378 0 0 14 14 active+clean 2021-05-12T12:08:15.448740+0000 31'100 61:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.055 INFO:tasks.workunit.client.0.smithi191.stdout:1.d 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.267160+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.055 INFO:tasks.workunit.client.0.smithi191.stdout:1.e 1 0 0 0 0 25 0 0 14 14 active+clean 2021-05-12T12:08:26.263024+0000 48'101 62:162 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.056 INFO:tasks.workunit.client.0.smithi191.stdout:1.f 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.348108+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.056 INFO:tasks.workunit.client.0.smithi191.stdout:1.24 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.097406+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.056 INFO:tasks.workunit.client.0.smithi191.stdout:1.25 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.265942+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.056 INFO:tasks.workunit.client.0.smithi191.stdout:1.26 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.265323+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.056 INFO:tasks.workunit.client.0.smithi191.stdout:1.27 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.262839+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.057 INFO:tasks.workunit.client.0.smithi191.stdout:1.28 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.096670+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.057 INFO:tasks.workunit.client.0.smithi191.stdout:1.29 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.266935+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.057 INFO:tasks.workunit.client.0.smithi191.stdout:1.2a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.270611+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.057 INFO:tasks.workunit.client.0.smithi191.stdout:1.2b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:15.516964+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.057 INFO:tasks.workunit.client.0.smithi191.stdout:1.2c 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:26.268151+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.058 INFO:tasks.workunit.client.0.smithi191.stdout:1.2d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.272722+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.058 INFO:tasks.workunit.client.0.smithi191.stdout:1.2e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.262059+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.058 INFO:tasks.workunit.client.0.smithi191.stdout:1.2f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.097618+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.058 INFO:tasks.workunit.client.0.smithi191.stdout:1.30 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096024+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.058 INFO:tasks.workunit.client.0.smithi191.stdout:1.31 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.004138+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.059 INFO:tasks.workunit.client.0.smithi191.stdout:1.32 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095890+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.059 INFO:tasks.workunit.client.0.smithi191.stdout:1.33 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.269201+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.059 INFO:tasks.workunit.client.0.smithi191.stdout:1.34 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.270232+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.059 INFO:tasks.workunit.client.0.smithi191.stdout:1.35 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:26.503487+0000 31'100 62:149 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:26.503429+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.059 INFO:tasks.workunit.client.0.smithi191.stdout:1.36 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266935+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.060 INFO:tasks.workunit.client.0.smithi191.stdout:1.37 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:28.711788+0000 31'100 62:154 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:28.711720+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.060 INFO:tasks.workunit.client.0.smithi191.stdout:1.38 1 0 0 0 0 377 0 0 10 10 active+clean 2021-05-12T12:08:15.447593+0000 31'100 61:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.060 INFO:tasks.workunit.client.0.smithi191.stdout:1.39 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096925+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.060 INFO:tasks.workunit.client.0.smithi191.stdout:1.3a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095289+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.060 INFO:tasks.workunit.client.0.smithi191.stdout:1.3b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.003550+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.061 INFO:tasks.workunit.client.0.smithi191.stdout:1.3c 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095256+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.061 INFO:tasks.workunit.client.0.smithi191.stdout:1.3d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.095883+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.061 INFO:tasks.workunit.client.0.smithi191.stdout:1.3e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.265635+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.061 INFO:tasks.workunit.client.0.smithi191.stdout:1.3f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.516826+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.061 INFO:tasks.workunit.client.0.smithi191.stdout:1.40 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.508296+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.062 INFO:tasks.workunit.client.0.smithi191.stdout:1.41 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.261407+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.062 INFO:tasks.workunit.client.0.smithi191.stdout:1.42 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096392+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.062 INFO:tasks.workunit.client.0.smithi191.stdout:1.43 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.265740+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.062 INFO:tasks.workunit.client.0.smithi191.stdout:1.44 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.264171+0000 31'100 62:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.062 INFO:tasks.workunit.client.0.smithi191.stdout:1.45 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:15.448214+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.063 INFO:tasks.workunit.client.0.smithi191.stdout:1.46 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.266582+0000 31'100 62:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.063 INFO:tasks.workunit.client.0.smithi191.stdout:1.47 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.093933+0000 31'100 62:144 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.063 INFO:tasks.workunit.client.0.smithi191.stdout:1.48 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.096751+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.063 INFO:tasks.workunit.client.0.smithi191.stdout:1.49 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.093204+0000 31'100 62:144 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.063 INFO:tasks.workunit.client.0.smithi191.stdout:1.4a 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-05-12T12:08:29.468800+0000 31'100 62:149 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:29.468737+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.064 INFO:tasks.workunit.client.0.smithi191.stdout:1.4b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:26.099050+0000 31'100 62:125 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.064 INFO:tasks.workunit.client.0.smithi191.stdout:1.4c 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:25.806920+0000 31'100 62:154 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:25.806836+0000 31'100 2021-05-12T12:08:25.806836+0000 0 2021-05-12T12:08:33.064 INFO:tasks.workunit.client.0.smithi191.stdout:1.4d 1 0 0 0 0 377 0 0 12 12 active+clean 2021-05-12T12:08:26.264985+0000 31'100 62:160 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:05:43.730688+0000 0'0 2021-05-12T12:05:43.730688+0000 0 2021-05-12T12:08:33.064 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:08:33.065 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:33.065 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:08:33.066 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2021-05-12T12:08:33.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2021-05-12T12:08:33.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:08:33.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T12:08:33.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:08:33.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T12:08:33.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:08:33.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T12:08:33.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:08:33.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T12:08:33.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2021-05-12T12:08:33.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:08:33.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2021-05-12T12:08:33.246 INFO:tasks.workunit.client.0.smithi191.stderr:error getting test/newobject_1: (2) No such file or directory 2021-05-12T12:08:33.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:08:33.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2021-05-12T12:08:33.275 INFO:tasks.workunit.client.0.smithi191.stderr:error getting test/newobject_2: (2) No such file or directory 2021-05-12T12:08:33.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2021-05-12T12:08:33.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-05-12T12:08:33.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:08:33.289 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:58.934+0000 7f1638f19700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 31'106 (31'4) delete 1:70b3bf12:::existing_4:head by client.4550.0:1 2021-05-12T12:06:13.073131+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:33.289 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:58.934+0000 7f1638f19700 20 _merge_object_divergent_entries: keeping 31'106 (31'4) delete 1:70b3bf12:::existing_4:head by client.4550.0:1 2021-05-12T12:06:13.073131+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:33.289 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:58.934+0000 7f1638f19700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:33.289 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:58.934+0000 7f1638f19700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 31'106 last_divergent_update: 31'106 2021-05-12T12:08:33.290 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:58.934+0000 7f1638f19700 10 _merge_object_divergent_entries: more recent entry found: 48'101 (31'4) modify 1:70b3bf12:::existing_4:head by client.4668.0:1 2021-05-12T12:07:14.440331+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2021-05-12T12:08:33.290 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 31'105 (31'3) delete 1:8881e6d8:::existing_3:head by client.4551.0:1 2021-05-12T12:06:13.073347+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:33.290 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 20 _merge_object_divergent_entries: keeping 31'105 (31'3) delete 1:8881e6d8:::existing_3:head by client.4551.0:1 2021-05-12T12:06:13.073347+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:33.290 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-05-12T12:08:33.291 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 31'105 last_divergent_update: 31'105 2021-05-12T12:08:33.291 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2021-05-12T12:08:33.291 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:08:33.291 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 31'106 2021-05-12T12:08:33.292 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 31'105 (31'3) delete 1:8881e6d8:::existing_3:head by client.4551.0:1 2021-05-12T12:06:13.073347+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:33.292 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-05-12T12:08:33.292 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.049+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'84 2021-05-12T12:08:33.292 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 31'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4545.0:1 2021-05-12T12:06:13.067710+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.292 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 20 _merge_object_divergent_entries: keeping 31'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4545.0:1 2021-05-12T12:06:13.067710+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.293 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:33.293 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 31'104 last_divergent_update: 31'104 2021-05-12T12:08:33.293 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2021-05-12T12:08:33.293 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-05-12T12:08:33.293 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 31'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4544.0:1 2021-05-12T12:06:13.067139+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.294 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 20 _merge_object_divergent_entries: keeping 31'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4544.0:1 2021-05-12T12:06:13.067139+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.294 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:33.294 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 31'103 last_divergent_update: 31'103 2021-05-12T12:08:33.294 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2021-05-12T12:08:33.295 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.052+0000 7f163a71c700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-05-12T12:08:33.295 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 31'101 (31'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2021-05-12T12:06:13.054049+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.295 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 20 _merge_object_divergent_entries: keeping 31'101 (31'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2021-05-12T12:06:13.054049+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.295 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:33.295 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 31'101 last_divergent_update: 31'101 2021-05-12T12:08:33.296 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2021-05-12T12:08:33.296 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:08:33.296 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 31'106 2021-05-12T12:08:33.339 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 31'101 (31'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2021-05-12T12:06:13.054049+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.339 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-05-12T12:08:33.339 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.053+0000 7f1639f1b700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'84 2021-05-12T12:08:33.339 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4538.0:1 2021-05-12T12:06:13.055332+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.340 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 20 _merge_object_divergent_entries: keeping 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4538.0:1 2021-05-12T12:06:13.055332+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.340 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:33.340 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 31'102 last_divergent_update: 31'102 2021-05-12T12:08:33.340 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2021-05-12T12:08:33.340 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:08:33.341 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 31'106 2021-05-12T12:08:33.341 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4538.0:1 2021-05-12T12:06:13.055332+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:33.341 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-05-12T12:08:33.341 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:07:59.220+0000 7f163af1d700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'84 2021-05-12T12:08:33.342 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-05-12T12:08:33.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:08:33.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-05-12T12:08:33.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:08:33.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2021-05-12T12:08:33.360 INFO:tasks.workunit.client.0.smithi191.stdout:success 2021-05-12T12:08:33.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.34183.out 2021-05-12T12:08:33.361 INFO:tasks.workunit.client.0.smithi191.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.34183.out': No such file or directory 2021-05-12T12:08:33.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2021-05-12T12:08:33.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T12:08:33.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T12:08:34.080 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test' does not exist 2021-05-12T12:08:34.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2021-05-12T12:08:34.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:08:34.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:08:34.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:08:34.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:08:34.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:08:34.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:08:34.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T12:08:34.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:08:34.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:08:34.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:08:34.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:08:34.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:08:34.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:08:34.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:08:34.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:08:34.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:08:34.207 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:08:34.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:08:34.208 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:08:34.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:08:34.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:08:34.208 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:08:34.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:08:34.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:08:34.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:08:34.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:08:34.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:08:34.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:08:34.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:08:34.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:08:34.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:34.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:34.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34183 2021-05-12T12:08:34.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:08:34.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:08:34.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T12:08:34.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T12:08:34.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T12:08:34.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T12:08:34.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:08:34.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:08:34.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:08:34.247 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:08:34.247 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:08:34.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:08:34.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:08:34.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:08:34.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:08:34.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:08:34.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:08:34.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:08:34.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:08:34.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:08:34.251 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:08:34.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:08:34.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:08:34.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:08:34.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:08:34.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:08:34.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:08:34.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:08:34.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:08:34.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:34.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:34.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34183 2021-05-12T12:08:34.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:08:34.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:08:34.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T12:08:34.258 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:08:34.258 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:34.258 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:34.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34183 2021-05-12T12:08:34.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:08:34.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:08:34.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:08:34.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T12:08:34.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2021-05-12T12:08:34.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2021-05-12T12:08:34.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2021-05-12T12:08:34.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2021-05-12T12:08:34.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2021-05-12T12:08:34.261 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2021-05-12T12:08:34.262 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2021-05-12T12:08:34.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2021-05-12T12:08:34.262 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:08:34.263 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:08:34.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2021-05-12T12:08:34.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T12:08:34.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:08:34.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:08:34.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:08:34.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T12:08:34.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T12:08:34.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:08:34.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:34.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:34.654 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:34.655 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:34.655 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:34.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:34.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:08:34.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:08:34.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:08:34.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:08:34.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:08:34.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:08:34.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:08:34.916 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:08:34.916 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:08:34.916 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:08:34.916 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:08:34.917 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:34.917 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:34.917 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-mon.a.asok 2021-05-12T12:08:34.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:08:34.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.34183/ceph-mon.a.asok config get fsid 2021-05-12T12:08:35.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:08:35.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:08:35.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:08:35.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:08:35.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:08:35.010 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:08:35.010 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:08:35.011 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:08:35.011 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:08:35.011 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:35.011 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:35.011 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-mon.a.asok 2021-05-12T12:08:35.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:08:35.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.34183/ceph-mon.a.asok config get mon_host 2021-05-12T12:08:35.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2021-05-12T12:08:35.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T12:08:35.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:08:35.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:08:35.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:08:35.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T12:08:35.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:08:36.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:08:36.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:36.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:36.012 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:36.012 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:36.013 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:36.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:36.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:08:36.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:08:36.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-05-12T12:08:36.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2021-05-12T12:08:36.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:08:36.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:08:36.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:08:36.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:08:36.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:08:36.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:08:36.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:08:36.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:08:36.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:08:36.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:08:36.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:08:36.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:08:36.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:08:36.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:08:36.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:08:36.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:36.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:36.047 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:36.047 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:36.047 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:36.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:36.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:36.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:08:36.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:08:36.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:08:36.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:08:36.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:08:36.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:08:36.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:08:36.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:36.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:08:36.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:08:36.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:08:36.050 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:08:36.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=3164e607-150e-4cc6-bf11-92a6263c5c19 2021-05-12T12:08:36.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 3164e607-150e-4cc6-bf11-92a6263c5c19' 2021-05-12T12:08:36.051 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 3164e607-150e-4cc6-bf11-92a6263c5c19 2021-05-12T12:08:36.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:08:36.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBExZtgTiOYAxAAghLmOOn8MlGZ167+A/IF+A== 2021-05-12T12:08:36.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBExZtgTiOYAxAAghLmOOn8MlGZ167+A/IF+A=="}' 2021-05-12T12:08:36.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 3164e607-150e-4cc6-bf11-92a6263c5c19 -i td/divergent-priors/0/new.json 2021-05-12T12:08:36.371 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:08:36.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T12:08:36.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBExZtgTiOYAxAAghLmOOn8MlGZ167+A/IF+A== --osd-uuid 3164e607-150e-4cc6-bf11-92a6263c5c19 2021-05-12T12:08:36.413 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:36.410+0000 7f75b0cf20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:36.415 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:36.412+0000 7f75b0cf20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:36.416 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:36.414+0000 7f75b0cf20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:36.417 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:36.415+0000 7f75b0cf20c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:08:36.417 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:36.415+0000 7f75b0cf20c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:08:36.418 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:36.415+0000 7f75b0cf20c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:08:36.453 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:36.450+0000 7f75b0cf20c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T12:08:41.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T12:08:41.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:08:41.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:08:41.269 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:08:41.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:08:41.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:08:41.752 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:08:41.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:08:41.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:41.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:08:41.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:08:41.780 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:41.777+0000 7f62745830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:41.782 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:41.779+0000 7f62745830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:41.783 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:41.780+0000 7f62745830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:42.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:08:42.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:08:42.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:08:42.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:08:42.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:08:42.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:42.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:08:42.122 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:08:42.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:42.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:08:42.298 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:42.295+0000 7f62745830c0 -1 Falling back to public interface 2021-05-12T12:08:42.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:08:43.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:08:43.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:43.485 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:08:43.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:08:43.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:43.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:08:43.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:08:44.421 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:44.418+0000 7f62745830c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:08:44.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:08:44.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:44.859 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:08:44.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:08:44.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:44.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:08:45.276 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/74408,v1:127.0.0.1:6803/74408] [v2:127.0.0.1:6804/74408,v1:127.0.0.1:6805/74408] exists,up 3164e607-150e-4cc6-bf11-92a6263c5c19 2021-05-12T12:08:45.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:08:45.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:08:45.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:08:45.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-05-12T12:08:45.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2021-05-12T12:08:45.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:08:45.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:08:45.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:08:45.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:08:45.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:08:45.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:08:45.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:08:45.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:08:45.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:08:45.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:08:45.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:08:45.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:08:45.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:08:45.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:08:45.293 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:08:45.293 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:45.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:45.294 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:45.295 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:45.295 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:45.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:45.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:45.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:08:45.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:08:45.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:08:45.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:08:45.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:08:45.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:08:45.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:08:45.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:45.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:08:45.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:08:45.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:08:45.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:08:45.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=4249e897-1cf7-49ad-8d28-f401f2373f5a 2021-05-12T12:08:45.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 4249e897-1cf7-49ad-8d28-f401f2373f5a' 2021-05-12T12:08:45.303 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 4249e897-1cf7-49ad-8d28-f401f2373f5a 2021-05-12T12:08:45.304 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:08:45.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBNxZtgeHGiERAALvKEe38aJe47Ldi1RnhDJA== 2021-05-12T12:08:45.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBNxZtgeHGiERAALvKEe38aJe47Ldi1RnhDJA=="}' 2021-05-12T12:08:45.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 4249e897-1cf7-49ad-8d28-f401f2373f5a -i td/divergent-priors/1/new.json 2021-05-12T12:08:45.800 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:08:45.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T12:08:45.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBNxZtgeHGiERAALvKEe38aJe47Ldi1RnhDJA== --osd-uuid 4249e897-1cf7-49ad-8d28-f401f2373f5a 2021-05-12T12:08:45.843 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:45.840+0000 7fd1311120c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:45.858 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:45.855+0000 7fd1311120c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:45.859 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:45.856+0000 7fd1311120c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:45.860 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:45.857+0000 7fd1311120c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:08:45.860 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:45.857+0000 7fd1311120c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:08:45.861 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:45.857+0000 7fd1311120c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:08:45.925 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:45.922+0000 7fd1311120c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T12:08:50.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T12:08:50.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:08:50.538 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:08:50.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:08:50.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:08:51.034 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:08:51.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:08:51.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:51.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:08:51.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:08:51.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:08:51.062 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:51.060+0000 7f775c43c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:51.064 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:51.061+0000 7f775c43c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:51.066 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:51.063+0000 7f775c43c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:51.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:08:51.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:08:51.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:08:51.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:08:51.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:08:51.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:51.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:08:51.408 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:08:51.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:51.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:08:51.583 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:51.580+0000 7f775c43c0c0 -1 Falling back to public interface 2021-05-12T12:08:51.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:08:52.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:08:52.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:52.775 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:08:52.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:08:52.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:52.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:08:53.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:08:53.827 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:53.824+0000 7f775c43c0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:08:54.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:08:54.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:54.152 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:08:54.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:08:54.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:54.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:08:54.608 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/75793,v1:127.0.0.1:6811/75793] [v2:127.0.0.1:6812/75793,v1:127.0.0.1:6813/75793] exists,up 4249e897-1cf7-49ad-8d28-f401f2373f5a 2021-05-12T12:08:54.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:08:54.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:08:54.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:08:54.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-05-12T12:08:54.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2021-05-12T12:08:54.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:08:54.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:08:54.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:08:54.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:08:54.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:08:54.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:54.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:54.617 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:54.617 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:54.618 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:08:54.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:54.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:08:54.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:08:54.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:08:54.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:08:54.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:08:54.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:08:54.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:08:54.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:08:54.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:54.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:08:54.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:08:54.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:08:54.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:08:54.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=d6a6ae15-1b91-42f9-8ba3-5cd4469c3918 2021-05-12T12:08:54.621 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 d6a6ae15-1b91-42f9-8ba3-5cd4469c3918 2021-05-12T12:08:54.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 d6a6ae15-1b91-42f9-8ba3-5cd4469c3918' 2021-05-12T12:08:54.622 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:08:54.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBWxZtg815yJRAACvJkUbXh6VcFmDwBHtpNBA== 2021-05-12T12:08:54.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBWxZtg815yJRAACvJkUbXh6VcFmDwBHtpNBA=="}' 2021-05-12T12:08:54.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new d6a6ae15-1b91-42f9-8ba3-5cd4469c3918 -i td/divergent-priors/2/new.json 2021-05-12T12:08:55.129 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:08:55.141 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T12:08:55.141 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBWxZtg815yJRAACvJkUbXh6VcFmDwBHtpNBA== --osd-uuid d6a6ae15-1b91-42f9-8ba3-5cd4469c3918 2021-05-12T12:08:55.171 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:55.168+0000 7f036be780c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:55.173 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:55.170+0000 7f036be780c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:55.174 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:55.171+0000 7f036be780c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:55.175 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:55.172+0000 7f036be780c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:08:55.175 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:55.172+0000 7f036be780c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:08:55.176 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:55.172+0000 7f036be780c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:08:55.264 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:08:55.262+0000 7f036be780c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T12:09:00.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T12:09:00.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:09:00.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:09:00.366 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:09:00.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:09:00.859 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:09:00.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:09:00.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:09:00.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:09:00.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:09:00.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:09:00.887 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:09:00.885+0000 7f57f34720c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:00.889 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:09:00.887+0000 7f57f34720c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:00.891 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:09:00.888+0000 7f57f34720c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:01.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:09:01.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:09:01.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:09:01.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:09:01.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:09:01.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:01.261 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:09:01.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:09:01.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:01.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:09:01.414 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:09:01.411+0000 7f57f34720c0 -1 Falling back to public interface 2021-05-12T12:09:01.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:09:02.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:09:02.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:02.630 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:09:02.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:09:02.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:02.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:09:03.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:09:03.726 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:09:03.723+0000 7f57f34720c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:09:04.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:09:04.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:04.030 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:09:04.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:09:04.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:04.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:09:04.439 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/77178,v1:127.0.0.1:6819/77178] [v2:127.0.0.1:6820/77178,v1:127.0.0.1:6821/77178] exists,up d6a6ae15-1b91-42f9-8ba3-5cd4469c3918 2021-05-12T12:09:04.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:09:04.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:09:04.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:09:04.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2021-05-12T12:09:05.260 INFO:tasks.workunit.client.0.smithi191.stderr:noout is set 2021-05-12T12:09:05.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2021-05-12T12:09:05.968 INFO:tasks.workunit.client.0.smithi191.stderr:noin is set 2021-05-12T12:09:05.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2021-05-12T12:09:06.738 INFO:tasks.workunit.client.0.smithi191.stderr:nodown is set 2021-05-12T12:09:06.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2021-05-12T12:09:06.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: create_ec_pool: local pool_name=test 2021-05-12T12:09:06.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: shift 2021-05-12T12:09:06.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: create_ec_pool: local allow_overwrites=true 2021-05-12T12:09:06.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: shift 2021-05-12T12:09:06.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-05-12T12:09:07.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: create_pool test 1 1 erasure myprofile 2021-05-12T12:09:07.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-05-12T12:09:08.041 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test' already exists 2021-05-12T12:09:08.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:09:09.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: create_ec_pool: '[' true = true ']' 2021-05-12T12:09:09.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2021-05-12T12:09:09.804 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 allow_ec_overwrites to true 2021-05-12T12:09:09.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: wait_for_clean 2021-05-12T12:09:09.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:09:09.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:09:09.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:09:09.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:09:09.816 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:09:09.816 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:09:09.816 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:09:09.816 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:09:09.817 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:09:09.817 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:09:09.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:09:09.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:09:09.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:09:09.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:09:09.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:09:10.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:09:10.302 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:09:10.302 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:09:10.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:09:10.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:10.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:09:10.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836487 2021-05-12T12:09:10.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836487 2021-05-12T12:09:10.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487' 2021-05-12T12:09:10.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:10.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:09:10.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672966 2021-05-12T12:09:10.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672966 2021-05-12T12:09:10.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2021-05-12T12:09:10.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:10.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:09:10.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542149 2021-05-12T12:09:10.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542149 2021-05-12T12:09:10.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-60129542149' 2021-05-12T12:09:10.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:10.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-05-12T12:09:10.707 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:10.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:09:10.708 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836487 2021-05-12T12:09:10.708 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:10.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836487 2021-05-12T12:09:10.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-05-12T12:09:10.709 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836487 2021-05-12T12:09:10.709 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:11.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836487 -lt 21474836487 2021-05-12T12:09:11.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:11.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-05-12T12:09:11.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:11.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:09:11.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672966 2021-05-12T12:09:11.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:11.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672966 2021-05-12T12:09:11.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-05-12T12:09:11.076 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672966 2021-05-12T12:09:11.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:09:11.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672966 2021-05-12T12:09:11.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:11.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542149 2021-05-12T12:09:11.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:11.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:09:11.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542149 2021-05-12T12:09:11.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:11.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542149 2021-05-12T12:09:11.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542149' 2021-05-12T12:09:11.498 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542149 2021-05-12T12:09:11.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:09:11.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542147 -lt 60129542149 2021-05-12T12:09:11.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:09:12.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:09:12.864 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:09:13.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542149 -lt 60129542149 2021-05-12T12:09:13.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:09:13.241 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:13.241 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:13.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:09:13.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:09:13.721 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:09:13.721 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:09:13.721 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:09:13.721 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:09:13.721 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:09:13.722 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:09:14.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:09:14.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:09:14.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:14.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:14.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:09:14.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:09:14.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:09:14.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: create_ec_pool: return 0 2021-05-12T12:09:14.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2021-05-12T12:09:14.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:09:14.561 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:09:14.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:09:14.928 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:09:14.928 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:09:14.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:09:14.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:14.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:09:15.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836490 2021-05-12T12:09:15.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836490 2021-05-12T12:09:15.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490' 2021-05-12T12:09:15.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:15.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:09:15.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672969 2021-05-12T12:09:15.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672969 2021-05-12T12:09:15.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2021-05-12T12:09:15.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:15.215 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:09:15.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542152 2021-05-12T12:09:15.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542152 2021-05-12T12:09:15.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672969 2-60129542152' 2021-05-12T12:09:15.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:15.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-05-12T12:09:15.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:15.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:09:15.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836490 2021-05-12T12:09:15.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:15.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836490 2021-05-12T12:09:15.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-05-12T12:09:15.353 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836490 2021-05-12T12:09:15.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:15.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836490 2021-05-12T12:09:15.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:09:16.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:09:16.717 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:17.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836490 2021-05-12T12:09:17.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:17.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-05-12T12:09:17.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:17.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:09:17.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672969 2021-05-12T12:09:17.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:17.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672969 2021-05-12T12:09:17.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-05-12T12:09:17.098 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672969 2021-05-12T12:09:17.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:09:17.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672969 2021-05-12T12:09:17.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:17.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542152 2021-05-12T12:09:17.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:17.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:09:17.477 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542152 2021-05-12T12:09:17.477 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:17.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542152 2021-05-12T12:09:17.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2021-05-12T12:09:17.478 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542152 2021-05-12T12:09:17.479 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:09:17.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542152 -lt 60129542152 2021-05-12T12:09:17.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2021-05-12T12:09:17.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:09:17.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:09:17.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:09:17.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:09:17.846 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:09:17.846 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:09:17.846 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:09:17.846 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:09:17.846 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:09:17.847 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:09:17.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:09:17.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:09:17.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:09:17.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:09:17.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:09:18.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:09:18.300 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:09:18.301 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:09:18.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:09:18.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:18.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:09:18.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836493 2021-05-12T12:09:18.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836493 2021-05-12T12:09:18.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836493' 2021-05-12T12:09:18.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:18.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:09:18.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672971 2021-05-12T12:09:18.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672971 2021-05-12T12:09:18.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836493 1-42949672971' 2021-05-12T12:09:18.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:18.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:09:18.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542154 2021-05-12T12:09:18.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542154 2021-05-12T12:09:18.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-60129542154' 2021-05-12T12:09:18.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:18.796 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836493 2021-05-12T12:09:18.796 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:18.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:09:18.797 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836493 2021-05-12T12:09:18.797 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:18.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836493 2021-05-12T12:09:18.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2021-05-12T12:09:18.798 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836493 2021-05-12T12:09:18.799 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:19.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836493 -lt 21474836493 2021-05-12T12:09:19.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:19.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2021-05-12T12:09:19.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:19.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:09:19.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672971 2021-05-12T12:09:19.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:19.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672971 2021-05-12T12:09:19.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-05-12T12:09:19.165 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672971 2021-05-12T12:09:19.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:09:19.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672971 -lt 42949672971 2021-05-12T12:09:19.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:19.537 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542154 2021-05-12T12:09:19.537 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:19.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:09:19.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542154 2021-05-12T12:09:19.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:19.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542154 2021-05-12T12:09:19.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2021-05-12T12:09:19.540 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542154 2021-05-12T12:09:19.540 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:09:19.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542152 -lt 60129542154 2021-05-12T12:09:19.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:09:20.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:09:20.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:09:21.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542155 -lt 60129542154 2021-05-12T12:09:21.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:09:21.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:21.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:21.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:09:21.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:09:21.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:09:21.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:09:21.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:09:21.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:09:21.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:09:21.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:09:22.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:09:22.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:09:22.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:22.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:22.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:09:22.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:09:22.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:09:22.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2021-05-12T12:09:22.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2021-05-12T12:09:23.081 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:09:23.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2021-05-12T12:09:23.091 INFO:tasks.workunit.client.0.smithi191.stdout:primary and soon to be divergent is 1 2021-05-12T12:09:23.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2021-05-12T12:09:23.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:09:23.436 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:09:23.437 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:09:08.922034+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:09:23.437 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:09:23.437 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:09:23.438 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:09:23.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2021-05-12T12:09:23.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-05-12T12:09:23.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2021-05-12T12:09:23.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2021-05-12T12:09:23.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-05-12T12:09:23.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2021-05-12T12:09:23.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2021-05-12T12:09:23.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-05-12T12:09:23.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2021-05-12T12:09:23.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2021-05-12T12:09:23.451 INFO:tasks.workunit.client.0.smithi191.stdout:writing initial objects 2021-05-12T12:09:23.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2021-05-12T12:09:23.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2021-05-12T12:09:23.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:23.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2021-05-12T12:09:23.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:23.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2021-05-12T12:09:23.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:23.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2021-05-12T12:09:24.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:24.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2021-05-12T12:09:24.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:24.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2021-05-12T12:09:24.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:24.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2021-05-12T12:09:24.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:24.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2021-05-12T12:09:24.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:24.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2021-05-12T12:09:25.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:25.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2021-05-12T12:09:25.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:25.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2021-05-12T12:09:25.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:25.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2021-05-12T12:09:25.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:25.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2021-05-12T12:09:26.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:26.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2021-05-12T12:09:26.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:26.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2021-05-12T12:09:26.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:26.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2021-05-12T12:09:26.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:26.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2021-05-12T12:09:26.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:26.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2021-05-12T12:09:27.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:27.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2021-05-12T12:09:27.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:27.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2021-05-12T12:09:27.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:27.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2021-05-12T12:09:28.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:28.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2021-05-12T12:09:28.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:28.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2021-05-12T12:09:28.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:28.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2021-05-12T12:09:28.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:28.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2021-05-12T12:09:28.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:28.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2021-05-12T12:09:29.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:29.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2021-05-12T12:09:29.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:29.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2021-05-12T12:09:29.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:29.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2021-05-12T12:09:29.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:29.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2021-05-12T12:09:30.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:30.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2021-05-12T12:09:30.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:30.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2021-05-12T12:09:30.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:30.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2021-05-12T12:09:30.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:30.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2021-05-12T12:09:31.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:31.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2021-05-12T12:09:31.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:31.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2021-05-12T12:09:31.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:31.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2021-05-12T12:09:31.654 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:31.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2021-05-12T12:09:31.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:31.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2021-05-12T12:09:32.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:32.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2021-05-12T12:09:32.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:32.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2021-05-12T12:09:32.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:32.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2021-05-12T12:09:32.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:32.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2021-05-12T12:09:33.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:33.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2021-05-12T12:09:33.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:33.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2021-05-12T12:09:33.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:33.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2021-05-12T12:09:33.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:33.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2021-05-12T12:09:33.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:33.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2021-05-12T12:09:34.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:34.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2021-05-12T12:09:34.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:34.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2021-05-12T12:09:34.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:34.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2021-05-12T12:09:34.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:34.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2021-05-12T12:09:35.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:35.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2021-05-12T12:09:35.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:35.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2021-05-12T12:09:35.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:35.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2021-05-12T12:09:35.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:35.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2021-05-12T12:09:36.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:36.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2021-05-12T12:09:36.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:36.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2021-05-12T12:09:36.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:36.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2021-05-12T12:09:36.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:36.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2021-05-12T12:09:36.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:36.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2021-05-12T12:09:37.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:37.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2021-05-12T12:09:37.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:37.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2021-05-12T12:09:37.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:37.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2021-05-12T12:09:38.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:38.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2021-05-12T12:09:38.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:38.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2021-05-12T12:09:38.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:38.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2021-05-12T12:09:38.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:38.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2021-05-12T12:09:38.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:38.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2021-05-12T12:09:39.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:39.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2021-05-12T12:09:39.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:39.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2021-05-12T12:09:39.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:39.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2021-05-12T12:09:39.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:39.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2021-05-12T12:09:40.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:40.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2021-05-12T12:09:40.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:40.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2021-05-12T12:09:40.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:40.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2021-05-12T12:09:40.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:40.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2021-05-12T12:09:40.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:40.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2021-05-12T12:09:41.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:41.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2021-05-12T12:09:41.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:41.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2021-05-12T12:09:41.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:41.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2021-05-12T12:09:41.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:41.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2021-05-12T12:09:42.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:42.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2021-05-12T12:09:42.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:42.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2021-05-12T12:09:42.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:42.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2021-05-12T12:09:42.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:42.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2021-05-12T12:09:43.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2021-05-12T12:09:43.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2021-05-12T12:09:43.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2021-05-12T12:09:43.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2021-05-12T12:09:43.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2021-05-12T12:09:44.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:44.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2021-05-12T12:09:44.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:44.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2021-05-12T12:09:44.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:44.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2021-05-12T12:09:44.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:44.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2021-05-12T12:09:45.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:45.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2021-05-12T12:09:45.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:45.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2021-05-12T12:09:45.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:45.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2021-05-12T12:09:45.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:45.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2021-05-12T12:09:45.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:45.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-05-12T12:09:46.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:46.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-05-12T12:09:46.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:09:46.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2021-05-12T12:09:46.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:09:46.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:09:46.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:09:46.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:09:46.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:09:46.425 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:09:46.425 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:09:46.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:09:46.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:09:46.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:09:46.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:09:46.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:09:46.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:09:46.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:09:46.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:09:46.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:09:46.866 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:09:46.866 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:09:46.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:09:46.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:46.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:09:46.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836501 2021-05-12T12:09:46.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836501 2021-05-12T12:09:47.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836501' 2021-05-12T12:09:47.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:47.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:09:47.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672979 2021-05-12T12:09:47.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672979 2021-05-12T12:09:47.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836501 1-42949672979' 2021-05-12T12:09:47.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:47.133 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:09:47.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542162 2021-05-12T12:09:47.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542162 2021-05-12T12:09:47.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-60129542162' 2021-05-12T12:09:47.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:47.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836501 2021-05-12T12:09:47.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:47.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:09:47.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836501 2021-05-12T12:09:47.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:47.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836501 2021-05-12T12:09:47.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2021-05-12T12:09:47.271 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836501 2021-05-12T12:09:47.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:47.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836499 -lt 21474836501 2021-05-12T12:09:47.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:09:48.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:09:48.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:49.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836501 -lt 21474836501 2021-05-12T12:09:49.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:49.028 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2021-05-12T12:09:49.029 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:49.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:09:49.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672979 2021-05-12T12:09:49.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:49.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672979 2021-05-12T12:09:49.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2021-05-12T12:09:49.031 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672979 2021-05-12T12:09:49.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:09:49.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672979 -lt 42949672979 2021-05-12T12:09:49.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:49.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542162 2021-05-12T12:09:49.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:49.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:09:49.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542162 2021-05-12T12:09:49.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:49.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542162 2021-05-12T12:09:49.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2021-05-12T12:09:49.423 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542162 2021-05-12T12:09:49.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:09:49.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542162 -lt 60129542162 2021-05-12T12:09:49.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:09:49.812 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:49.812 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:50.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:09:50.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:09:50.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:09:50.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:09:50.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:09:50.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:09:50.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:09:50.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:09:50.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:09:50.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:09:50.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:50.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:51.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:09:51.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:09:51.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:09:51.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2021-05-12T12:09:51.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T12:09:51.128 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T12:09:51.128 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T12:09:51.128 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T12:09:51.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2021-05-12T12:09:51.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2021-05-12T12:09:51.494 INFO:tasks.workunit.client.0.smithi191.stdout:blackholing osds 0 2 2021-05-12T12:09:51.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:09:51.866 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:09:51.866 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T12:09:08.922034+0000 27'100 27:111 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:09:51.867 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:09:51.867 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:09:51.868 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:09:51.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:09:51.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2021-05-12T12:09:51.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:09:51.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:09:51.880 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:09:51.880 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:09:51.880 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:09:51.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.0.asok 2021-05-12T12:09:51.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:09:51.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T12:09:51.970 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:09:51.970 INFO:tasks.workunit.client.0.smithi191.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:09:51.970 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:09:51.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:09:51.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2021-05-12T12:09:51.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:09:51.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:09:51.980 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:09:51.980 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:09:51.980 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:09:51.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.2.asok 2021-05-12T12:09:51.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:09:51.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.2.asok config set objectstore_blackhole 1 2021-05-12T12:09:52.069 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:09:52.070 INFO:tasks.workunit.client.0.smithi191.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:09:52.070 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:09:52.077 INFO:tasks.workunit.client.0.smithi191.stdout:writing divergent object 2021-05-12T12:09:52.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2021-05-12T12:09:52.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2021-05-12T12:09:52.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2021-05-12T12:09:53.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2021-05-12T12:09:53.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-05-12T12:09:53.315 INFO:tasks.workunit.client.0.smithi191.stdout:created pool test snap snap1 2021-05-12T12:09:53.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2021-05-12T12:09:53.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2021-05-12T12:09:53.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-05-12T12:10:03.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2021-05-12T12:10:03.321 INFO:tasks.workunit.client.0.smithi191.stderr:rados: no process found 2021-05-12T12:10:03.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2021-05-12T12:10:03.321 INFO:tasks.workunit.client.0.smithi191.stdout:killing all the osds 2021-05-12T12:10:03.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:10:03.671 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:10:03.671 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2021-05-12T12:09:08.922034+0000 28'104 28:115 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:10:03.671 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:10:03.671 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:10:03.672 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:10:03.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2021-05-12T12:10:03.683 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:10:03.683 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:10:03.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:10:03.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:10:03.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:10:03.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:10:03.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-05-12T12:10:03.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2021-05-12T12:10:04.322 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already down. 2021-05-12T12:10:04.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-05-12T12:10:04.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2021-05-12T12:10:04.838 INFO:tasks.workunit.client.0.smithi191.stderr:osd.1 is already down. 2021-05-12T12:10:04.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-05-12T12:10:04.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2021-05-12T12:10:05.350 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already down. 2021-05-12T12:10:05.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:05.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2021-05-12T12:10:05.934 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already out. 2021-05-12T12:10:05.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:05.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2021-05-12T12:10:06.484 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already out. 2021-05-12T12:10:06.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2021-05-12T12:10:06.494 INFO:tasks.workunit.client.0.smithi191.stdout:bringing up non_divergent 0 2 2021-05-12T12:10:06.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:10:06.841 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:10:06.841 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2021-05-12T12:09:08.922034+0000 28'104 28:115 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:10:06.842 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:10:06.842 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:10:06.843 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:10:06.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:06.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-05-12T12:10:06.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:10:06.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:10:06.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:10:06.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:10:06.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:10:06.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:10:06.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:10:06.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:10:06.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:10:06.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:10:06.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:10:06.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:10:06.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:10:06.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:10:06.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:10:06.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:10:06.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:10:06.860 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:10:06.860 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:10:06.860 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:10:06.860 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:10:06.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:10:06.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:10:06.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:10:06.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:10:06.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:10:06.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:10:06.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:10:06.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:10:06.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:10:06.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:10:06.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:10:06.863 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:10:06.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:10:06.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:10:06.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:10:06.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:10:06.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:10:06.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:10:06.883 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:06.880+0000 7f15652660c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:06.885 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:06.882+0000 7f15652660c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:06.887 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:06.883+0000 7f15652660c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:07.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:10:07.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:10:07.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:10:07.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:10:07.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:10:07.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:07.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:10:07.247 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:10:07.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:07.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:10:07.403 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:07.400+0000 7f15652660c0 -1 Falling back to public interface 2021-05-12T12:10:07.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:08.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:08.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:08.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:10:08.615 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:10:08.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:08.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:10:09.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:10.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:10.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:10.004 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:10:10.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:10:10.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:10.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:10:10.156 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:10.153+0000 7f15652660c0 -1 osd.0 27 log_to_monitors {default=true} 2021-05-12T12:10:10.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:11.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:11.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:11.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:10:11.423 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:10:11.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:11.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:10:11.791 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [5,27) [v2:127.0.0.1:6802/81623,v1:127.0.0.1:6803/81623] [v2:127.0.0.1:6804/81623,v1:127.0.0.1:6805/81623] exists,up 3164e607-150e-4cc6-bf11-92a6263c5c19 2021-05-12T12:10:11.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:10:11.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:10:11.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:10:11.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:11.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-05-12T12:10:11.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:10:11.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:10:11.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:10:11.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:10:11.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:10:11.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:10:11.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:10:11.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:10:11.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:10:11.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:10:11.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:10:11.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:10:11.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:10:11.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:10:11.799 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:10:11.799 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:10:11.799 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:10:11.799 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:10:11.800 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:10:11.800 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:10:11.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:10:11.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:10:11.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:10:11.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:10:11.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:10:11.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:10:11.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:10:11.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:10:11.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:10:11.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:10:11.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:10:11.802 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:10:11.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:10:11.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:10:11.804 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:10:11.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:10:11.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:10:11.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:10:11.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:10:11.821 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:11.818+0000 7f39c69f70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:11.823 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:11.820+0000 7f39c69f70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:11.825 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:11.821+0000 7f39c69f70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:12.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:10:12.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:10:12.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:10:12.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:10:12.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:10:12.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:12.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:10:12.179 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:10:12.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:12.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:10:12.344 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:12.342+0000 7f39c69f70c0 -1 Falling back to public interface 2021-05-12T12:10:12.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:13.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:13.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:13.546 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:10:13.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:10:13.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:13.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:10:13.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:14.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:14.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:14.924 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:10:14.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:10:14.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:14.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:10:15.201 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:10:15.198+0000 7f39c69f70c0 -1 osd.2 27 log_to_monitors {default=true} 2021-05-12T12:10:15.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:16.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:16.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:16.303 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:10:16.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:10:16.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:16.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:10:16.678 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [14,27) [v2:127.0.0.1:6810/82206,v1:127.0.0.1:6811/82206] [v2:127.0.0.1:6812/82206,v1:127.0.0.1:6813/82206] exists,up d6a6ae15-1b91-42f9-8ba3-5cd4469c3918 2021-05-12T12:10:16.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:10:16.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:10:16.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:10:16.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:16.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2021-05-12T12:10:17.245 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already in. 2021-05-12T12:10:17.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:17.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2021-05-12T12:10:17.811 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already in. 2021-05-12T12:10:17.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2021-05-12T12:10:22.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2021-05-12T12:10:22.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2021-05-12T12:10:22.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2021-05-12T12:10:22.824 INFO:tasks.workunit.client.0.smithi191.stdout:writing non-divergent object existing_4 2021-05-12T12:10:22.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:10:23.177 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:10:23.177 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:10:18.265784+0000 27'100 40:120 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:10:23.177 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:10:23.178 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:10:23.179 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:10:23.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T12:10:23.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:10:23.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2021-05-12T12:10:23.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:10:23.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:10:23.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:10:23.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:10:23.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:10:23.319 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:10:23.319 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:10:23.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:10:23.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:10:23.321 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:10:23.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:10:23.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:10:23.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:10:23.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:10:23.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:10:23.739 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:10:23.739 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:10:23.740 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:10:23.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:10:23.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:10:23.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:10:23.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=150323855365 2021-05-12T12:10:23.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 150323855365 2021-05-12T12:10:23.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-150323855365' 2021-05-12T12:10:23.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:10:23.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:10:23.972 INFO:tasks.workunit.client.0.smithi191.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:10:23.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:10:23.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:10:23.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:10:23.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:10:23.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:10:24.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=158913789956 2021-05-12T12:10:24.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 158913789956 2021-05-12T12:10:24.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-150323855365 2-158913789956' 2021-05-12T12:10:24.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:10:24.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-150323855365 2021-05-12T12:10:24.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:10:24.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:10:24.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-150323855365 2021-05-12T12:10:24.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:10:24.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=150323855365 2021-05-12T12:10:24.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2021-05-12T12:10:24.112 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 150323855365 2021-05-12T12:10:24.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:10:24.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 150323855363 -lt 150323855365 2021-05-12T12:10:24.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:10:25.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:10:25.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:10:25.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 150323855365 -lt 150323855365 2021-05-12T12:10:25.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:10:25.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-158913789956 2021-05-12T12:10:25.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:10:25.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:10:25.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-158913789956 2021-05-12T12:10:25.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:10:25.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=158913789956 2021-05-12T12:10:25.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 158913789956' 2021-05-12T12:10:25.872 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 158913789956 2021-05-12T12:10:25.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:10:26.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 158913789956 -lt 158913789956 2021-05-12T12:10:26.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:10:26.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:26.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:26.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:10:26.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:26.722 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:26.722 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:26.722 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:26.723 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:26.723 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:26.723 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:27.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:27.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:27.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:27.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:27.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:27.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:10:27.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:10:27.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:10:27.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:27.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:10:27.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:27.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:27.665 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:27.665 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:27.665 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:27.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:27.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:27.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:28.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:28.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:28.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:28.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:28.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:28.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:28.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:28.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:28.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:28.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:28.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:28.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:28.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:29.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:29.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:29.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:10:29.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:29.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:10:29.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:29.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:29.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:29.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:29.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:29.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:29.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:29.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:29.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:29.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:29.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:29.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:30.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:30.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:30.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:30.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:30.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:30.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:30.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:30.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:30.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:30.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:30.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:30.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:10:30.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:30.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:10:30.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:30.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:30.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:30.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:30.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:30.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:30.942 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:30.942 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:31.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:31.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:31.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:31.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:31.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:31.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:31.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:31.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:31.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:31.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:31.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:31.777 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:31.778 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:32.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:32.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:32.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:10:32.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:32.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:10:33.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:33.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:33.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:33.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:33.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:33.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:33.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:33.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:33.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:33.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:33.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:33.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:33.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:33.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:33.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:33.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:33.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:33.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:33.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:33.900 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:33.900 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:34.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:34.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:34.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:10:34.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:34.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:10:35.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:35.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:35.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:35.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:35.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:35.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:35.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:35.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:36.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:36.346 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:36.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:36.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:36.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:36.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:36.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:36.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:36.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:36.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:36.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:36.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:36.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:37.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:37.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:37.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:10:37.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:37.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:10:40.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:40.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:40.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:40.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:40.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:40.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:40.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:40.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:40.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:40.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:40.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:40.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:41.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:41.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:41.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:41.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:41.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:41.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:41.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:41.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:41.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:41.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:41.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:41.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:10:41.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:41.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:10:48.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:48.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:48.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:48.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:48.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:48.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:48.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:48.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:48.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:48.622 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:48.622 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:48.622 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:49.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:49.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:49.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:49.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:49.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:49.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:49.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:49.138 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:49.139 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:49.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:49.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:49.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:10:49.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:49.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:10:56.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:56.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:56.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:56.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:56.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:56.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:56.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:56.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:57.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:57.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:57.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:57.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:57.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:57.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:57.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:57.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:57.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:57.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:57.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:57.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:57.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:58.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:58.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:58.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:10:58.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:10:58.709 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:10:58.709 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_fingerprint": "7bf9d246-45ca-4870-98ba-18a9f48e5f82", 2021-05-12T12:10:58.709 INFO:tasks.workunit.client.0.smithi191.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:10:58.710 INFO:tasks.workunit.client.0.smithi191.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:10:58.710 INFO:tasks.workunit.client.0.smithi191.stdout: "timestamp": "2021-05-12T12:10:58.705573+0000", 2021-05-12T12:10:58.710 INFO:tasks.workunit.client.0.smithi191.stdout: "tag": "", 2021-05-12T12:10:58.710 INFO:tasks.workunit.client.0.smithi191.stdout: "health": { 2021-05-12T12:10:58.710 INFO:tasks.workunit.client.0.smithi191.stdout: "status": "HEALTH_ERR", 2021-05-12T12:10:58.711 INFO:tasks.workunit.client.0.smithi191.stdout: "checks": { 2021-05-12T12:10:58.711 INFO:tasks.workunit.client.0.smithi191.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:10:58.711 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:10:58.711 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:10:58.711 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:10:58.712 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:10:58.712 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.712 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:10:58.712 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.712 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:10:58.713 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.713 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.713 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:10:58.713 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.713 INFO:tasks.workunit.client.0.smithi191.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:10:58.714 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:10:58.714 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:10:58.714 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:10:58.714 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 17 2021-05-12T12:10:58.714 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.715 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [], 2021-05-12T12:10:58.715 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:10:58.715 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.715 INFO:tasks.workunit.client.0.smithi191.stdout: "OSD_DOWN": { 2021-05-12T12:10:58.715 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:10:58.716 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:10:58.716 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 osds down", 2021-05-12T12:10:58.716 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:10:58.716 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.716 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:10:58.717 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.717 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "osd.1 (root=default,host=smithi191) is down" 2021-05-12T12:10:58.717 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.717 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.718 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:10:58.718 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.718 INFO:tasks.workunit.client.0.smithi191.stdout: "PG_DEGRADED": { 2021-05-12T12:10:58.718 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:10:58.718 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:10:58.719 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2021-05-12T12:10:58.719 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:10:58.719 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.719 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:10:58.720 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.720 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2021-05-12T12:10:58.720 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.720 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.721 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:10:58.721 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.721 INFO:tasks.workunit.client.0.smithi191.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:10:58.721 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:10:58.722 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:10:58.722 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:10:58.722 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:10:58.722 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.722 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:10:58.723 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.723 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:10:58.723 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.723 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.723 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:10:58.724 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.724 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.724 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:10:58.724 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.724 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.724 INFO:tasks.workunit.client.0.smithi191.stdout: "mutes": [] 2021-05-12T12:10:58.725 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.725 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_first_committed": 1, 2021-05-12T12:10:58.725 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_last_committed": 1, 2021-05-12T12:10:58.725 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap": { 2021-05-12T12:10:58.725 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:10:58.726 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:10:58.726 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:08:34.279716Z", 2021-05-12T12:10:58.726 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:08:34.279716Z", 2021-05-12T12:10:58.726 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release": 17, 2021-05-12T12:10:58.727 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:10:58.727 INFO:tasks.workunit.client.0.smithi191.stdout: "election_strategy": 1, 2021-05-12T12:10:58.727 INFO:tasks.workunit.client.0.smithi191.stdout: "disallowed_leaders: ": "", 2021-05-12T12:10:58.727 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": false, 2021-05-12T12:10:58.727 INFO:tasks.workunit.client.0.smithi191.stdout: "features": { 2021-05-12T12:10:58.728 INFO:tasks.workunit.client.0.smithi191.stdout: "persistent": [ 2021-05-12T12:10:58.728 INFO:tasks.workunit.client.0.smithi191.stdout: "kraken", 2021-05-12T12:10:58.728 INFO:tasks.workunit.client.0.smithi191.stdout: "luminous", 2021-05-12T12:10:58.728 INFO:tasks.workunit.client.0.smithi191.stdout: "mimic", 2021-05-12T12:10:58.728 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap-prune", 2021-05-12T12:10:58.729 INFO:tasks.workunit.client.0.smithi191.stdout: "nautilus", 2021-05-12T12:10:58.729 INFO:tasks.workunit.client.0.smithi191.stdout: "octopus", 2021-05-12T12:10:58.729 INFO:tasks.workunit.client.0.smithi191.stdout: "pacific", 2021-05-12T12:10:58.729 INFO:tasks.workunit.client.0.smithi191.stdout: "elector-pinging", 2021-05-12T12:10:58.730 INFO:tasks.workunit.client.0.smithi191.stdout: "quincy" 2021-05-12T12:10:58.730 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.730 INFO:tasks.workunit.client.0.smithi191.stdout: "optional": [] 2021-05-12T12:10:58.731 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.731 INFO:tasks.workunit.client.0.smithi191.stdout: "mons": [ 2021-05-12T12:10:58.731 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.732 INFO:tasks.workunit.client.0.smithi191.stdout: "rank": 0, 2021-05-12T12:10:58.732 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "a", 2021-05-12T12:10:58.732 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:10:58.732 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.733 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.733 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.733 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:10:58.733 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 0 2021-05-12T12:10:58.733 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.734 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.734 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.734 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:10:58.734 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:10:58.735 INFO:tasks.workunit.client.0.smithi191.stdout: "priority": 0, 2021-05-12T12:10:58.735 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 0, 2021-05-12T12:10:58.735 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_location": "{}" 2021-05-12T12:10:58.735 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.735 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.736 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.736 INFO:tasks.workunit.client.0.smithi191.stdout: "quorum": [ 2021-05-12T12:10:58.736 INFO:tasks.workunit.client.0.smithi191.stdout: 0 2021-05-12T12:10:58.736 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.736 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap": { 2021-05-12T12:10:58.737 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 40, 2021-05-12T12:10:58.737 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:10:58.737 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:08:35.207927+0000", 2021-05-12T12:10:58.737 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:10:17.913873+0000", 2021-05-12T12:10:58.737 INFO:tasks.workunit.client.0.smithi191.stdout: "last_up_change": "2021-05-12T12:10:15.398106+0000", 2021-05-12T12:10:58.738 INFO:tasks.workunit.client.0.smithi191.stdout: "last_in_change": "2021-05-12T12:10:17.579142+0000", 2021-05-12T12:10:58.738 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:10:58.738 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_num": 5800384, 2021-05-12T12:10:58.738 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_set": [ 2021-05-12T12:10:58.738 INFO:tasks.workunit.client.0.smithi191.stdout: "nodown", 2021-05-12T12:10:58.739 INFO:tasks.workunit.client.0.smithi191.stdout: "noin", 2021-05-12T12:10:58.739 INFO:tasks.workunit.client.0.smithi191.stdout: "noout", 2021-05-12T12:10:58.739 INFO:tasks.workunit.client.0.smithi191.stdout: "pglog_hardlimit", 2021-05-12T12:10:58.739 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snapdirs", 2021-05-12T12:10:58.739 INFO:tasks.workunit.client.0.smithi191.stdout: "recovery_deletes", 2021-05-12T12:10:58.740 INFO:tasks.workunit.client.0.smithi191.stdout: "sortbitwise" 2021-05-12T12:10:58.740 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.740 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_version": 10, 2021-05-12T12:10:58.740 INFO:tasks.workunit.client.0.smithi191.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:10:58.740 INFO:tasks.workunit.client.0.smithi191.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:10:58.741 INFO:tasks.workunit.client.0.smithi191.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:10:58.741 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_snapshot": "", 2021-05-12T12:10:58.741 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_max": 1, 2021-05-12T12:10:58.741 INFO:tasks.workunit.client.0.smithi191.stdout: "max_osd": 3, 2021-05-12T12:10:58.741 INFO:tasks.workunit.client.0.smithi191.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:10:58.742 INFO:tasks.workunit.client.0.smithi191.stdout: "min_compat_client": "jewel", 2021-05-12T12:10:58.742 INFO:tasks.workunit.client.0.smithi191.stdout: "require_osd_release": "quincy", 2021-05-12T12:10:58.742 INFO:tasks.workunit.client.0.smithi191.stdout: "pools": [ 2021-05-12T12:10:58.742 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.742 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:10:58.743 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_name": "test", 2021-05-12T12:10:58.743 INFO:tasks.workunit.client.0.smithi191.stdout: "create_time": "2021-05-12T12:09:07.671909+0000", 2021-05-12T12:10:58.743 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": 16389, 2021-05-12T12:10:58.743 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-05-12T12:10:58.743 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 3, 2021-05-12T12:10:58.744 INFO:tasks.workunit.client.0.smithi191.stdout: "size": 3, 2021-05-12T12:10:58.744 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 2, 2021-05-12T12:10:58.744 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_rule": 1, 2021-05-12T12:10:58.744 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:10:58.744 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:10:58.745 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:10:58.745 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:10:58.745 INFO:tasks.workunit.client.0.smithi191.stdout: "object_hash": 2, 2021-05-12T12:10:58.745 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:10:58.745 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num": 1, 2021-05-12T12:10:58.746 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num": 1, 2021-05-12T12:10:58.746 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num_target": 1, 2021-05-12T12:10:58.746 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_target": 1, 2021-05-12T12:10:58.746 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_pending": 1, 2021-05-12T12:10:58.746 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pg_merge_meta": { 2021-05-12T12:10:58.747 INFO:tasks.workunit.client.0.smithi191.stdout: "source_pgid": "0.0", 2021-05-12T12:10:58.747 INFO:tasks.workunit.client.0.smithi191.stdout: "ready_epoch": 0, 2021-05-12T12:10:58.747 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_started": 0, 2021-05-12T12:10:58.747 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": 0, 2021-05-12T12:10:58.748 INFO:tasks.workunit.client.0.smithi191.stdout: "source_version": "0'0", 2021-05-12T12:10:58.748 INFO:tasks.workunit.client.0.smithi191.stdout: "target_version": "0'0" 2021-05-12T12:10:58.748 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.748 INFO:tasks.workunit.client.0.smithi191.stdout: "last_change": "28", 2021-05-12T12:10:58.748 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend": "0", 2021-05-12T12:10:58.749 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:10:58.749 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:10:58.749 INFO:tasks.workunit.client.0.smithi191.stdout: "auid": 0, 2021-05-12T12:10:58.749 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_mode": "pool", 2021-05-12T12:10:58.750 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_seq": 1, 2021-05-12T12:10:58.750 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_epoch": 28, 2021-05-12T12:10:58.750 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_snaps": [ 2021-05-12T12:10:58.750 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.751 INFO:tasks.workunit.client.0.smithi191.stdout: "snapid": 1, 2021-05-12T12:10:58.751 INFO:tasks.workunit.client.0.smithi191.stdout: "stamp": "2021-05-12T12:09:53.148800+0000", 2021-05-12T12:10:58.751 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "snap1" 2021-05-12T12:10:58.751 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.752 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.752 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps": "[]", 2021-05-12T12:10:58.752 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_bytes": 0, 2021-05-12T12:10:58.752 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_objects": 0, 2021-05-12T12:10:58.752 INFO:tasks.workunit.client.0.smithi191.stdout: "tiers": [], 2021-05-12T12:10:58.753 INFO:tasks.workunit.client.0.smithi191.stdout: "tier_of": -1, 2021-05-12T12:10:58.753 INFO:tasks.workunit.client.0.smithi191.stdout: "read_tier": -1, 2021-05-12T12:10:58.753 INFO:tasks.workunit.client.0.smithi191.stdout: "write_tier": -1, 2021-05-12T12:10:58.753 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_mode": "none", 2021-05-12T12:10:58.754 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_bytes": 0, 2021-05-12T12:10:58.754 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_objects": 0, 2021-05-12T12:10:58.754 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:10:58.754 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:10:58.754 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:10:58.755 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_flush_age": 0, 2021-05-12T12:10:58.755 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_evict_age": 0, 2021-05-12T12:10:58.755 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profile": "myprofile", 2021-05-12T12:10:58.755 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_params": { 2021-05-12T12:10:58.755 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "none" 2021-05-12T12:10:58.756 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.756 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_period": 0, 2021-05-12T12:10:58.756 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_count": 0, 2021-05-12T12:10:58.756 INFO:tasks.workunit.client.0.smithi191.stdout: "use_gmt_hitset": true, 2021-05-12T12:10:58.757 INFO:tasks.workunit.client.0.smithi191.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:10:58.757 INFO:tasks.workunit.client.0.smithi191.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:10:58.757 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:10:58.757 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:10:58.757 INFO:tasks.workunit.client.0.smithi191.stdout: "grade_table": [], 2021-05-12T12:10:58.758 INFO:tasks.workunit.client.0.smithi191.stdout: "stripe_width": 8192, 2021-05-12T12:10:58.758 INFO:tasks.workunit.client.0.smithi191.stdout: "expected_num_objects": 0, 2021-05-12T12:10:58.758 INFO:tasks.workunit.client.0.smithi191.stdout: "fast_read": false, 2021-05-12T12:10:58.758 INFO:tasks.workunit.client.0.smithi191.stdout: "options": {}, 2021-05-12T12:10:58.759 INFO:tasks.workunit.client.0.smithi191.stdout: "application_metadata": {} 2021-05-12T12:10:58.759 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.759 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.759 INFO:tasks.workunit.client.0.smithi191.stdout: "osds": [ 2021-05-12T12:10:58.760 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.760 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:10:58.760 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "3164e607-150e-4cc6-bf11-92a6263c5c19", 2021-05-12T12:10:58.760 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:10:58.760 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:10:58.761 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:10:58.761 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:10:58.761 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 5, 2021-05-12T12:10:58.761 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 27, 2021-05-12T12:10:58.761 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 35, 2021-05-12T12:10:58.762 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 39, 2021-05-12T12:10:58.762 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 29, 2021-05-12T12:10:58.762 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:10:58.763 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:10:58.763 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.763 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.763 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.763 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:10:58.764 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 81623 2021-05-12T12:10:58.764 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.764 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.764 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.764 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:10:58.764 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 81623 2021-05-12T12:10:58.765 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.765 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.765 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.765 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:10:58.766 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.766 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.766 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.766 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:10:58.766 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 81623 2021-05-12T12:10:58.767 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.767 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.767 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.767 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:10:58.767 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 81623 2021-05-12T12:10:58.768 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.768 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.768 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.768 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:10:58.768 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.769 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.769 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.769 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:10:58.769 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 81623 2021-05-12T12:10:58.769 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.770 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.770 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.770 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:10:58.770 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 81623 2021-05-12T12:10:58.771 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.771 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.771 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.771 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:10:58.771 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.772 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.772 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.772 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:10:58.772 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 81623 2021-05-12T12:10:58.772 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.773 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.773 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.773 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:10:58.773 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 81623 2021-05-12T12:10:58.773 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.774 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.774 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.774 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6803/81623", 2021-05-12T12:10:58.774 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6805/81623", 2021-05-12T12:10:58.774 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6809/81623", 2021-05-12T12:10:58.775 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6807/81623", 2021-05-12T12:10:58.775 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:10:58.775 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:10:58.775 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:10:58.775 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.775 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.776 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.776 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:10:58.776 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "4249e897-1cf7-49ad-8d28-f401f2373f5a", 2021-05-12T12:10:58.776 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 0, 2021-05-12T12:10:58.776 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:10:58.777 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:10:58.777 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:10:58.777 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 0, 2021-05-12T12:10:58.777 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 0, 2021-05-12T12:10:58.777 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 10, 2021-05-12T12:10:58.778 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 23, 2021-05-12T12:10:58.778 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 30, 2021-05-12T12:10:58.778 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:10:58.778 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:10:58.778 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.779 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.779 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.779 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:10:58.779 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:10:58.779 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.780 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.780 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.780 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:10:58.780 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:10:58.780 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.781 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.781 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.781 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:10:58.781 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.782 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.782 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.782 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:10:58.782 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:10:58.782 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.783 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.783 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.783 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:10:58.783 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:10:58.783 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.783 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.784 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.784 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:10:58.784 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.784 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.785 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.785 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:10:58.785 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:10:58.785 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.785 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.786 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.786 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:10:58.786 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:10:58.786 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.786 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.787 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.787 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:10:58.787 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.787 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.787 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.788 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:10:58.788 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:10:58.788 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.788 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.788 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.789 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:10:58.789 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:10:58.789 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.789 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.789 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.790 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/75793", 2021-05-12T12:10:58.790 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/75793", 2021-05-12T12:10:58.790 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/75793", 2021-05-12T12:10:58.790 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/75793", 2021-05-12T12:10:58.791 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:10:58.791 INFO:tasks.workunit.client.0.smithi191.stdout: "exists" 2021-05-12T12:10:58.791 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.791 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.791 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.792 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:10:58.792 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "d6a6ae15-1b91-42f9-8ba3-5cd4469c3918", 2021-05-12T12:10:58.792 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:10:58.792 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:10:58.792 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:10:58.793 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:10:58.793 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 14, 2021-05-12T12:10:58.793 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 27, 2021-05-12T12:10:58.793 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 37, 2021-05-12T12:10:58.794 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 0, 2021-05-12T12:10:58.794 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 31, 2021-05-12T12:10:58.794 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:10:58.794 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:10:58.794 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.795 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.795 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.795 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:10:58.795 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 82206 2021-05-12T12:10:58.795 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.796 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.796 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.796 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:10:58.796 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 82206 2021-05-12T12:10:58.797 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.797 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.797 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.797 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:10:58.797 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.798 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.798 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.798 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:10:58.798 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 82206 2021-05-12T12:10:58.799 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.799 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.799 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.799 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:10:58.800 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 82206 2021-05-12T12:10:58.800 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.800 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.800 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.800 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:10:58.801 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.801 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.801 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.801 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:10:58.801 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 82206 2021-05-12T12:10:58.802 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.802 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.802 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.802 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:10:58.803 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 82206 2021-05-12T12:10:58.803 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.803 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.803 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.803 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:10:58.804 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:10:58.804 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.804 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:10:58.804 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:10:58.805 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 82206 2021-05-12T12:10:58.805 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.805 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.805 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:10:58.805 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:10:58.806 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 82206 2021-05-12T12:10:58.806 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.806 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.806 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.807 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/82206", 2021-05-12T12:10:58.807 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/82206", 2021-05-12T12:10:58.807 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/82206", 2021-05-12T12:10:58.807 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/82206", 2021-05-12T12:10:58.807 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:10:58.808 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:10:58.808 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:10:58.808 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.808 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.808 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.809 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_xinfo": [ 2021-05-12T12:10:58.809 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.809 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:10:58.809 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:10:04.097634+0000", 2021-05-12T12:10:58.810 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:10:58.810 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:10:58.810 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:10:58.810 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:10:58.810 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:08:45.451069+0000", 2021-05-12T12:10:58.811 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:10:58.811 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.811 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.811 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:10:58.811 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:10:04.641605+0000", 2021-05-12T12:10:58.812 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:10:58.812 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:10:58.812 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:10:58.812 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:10:58.812 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:08:54.873051+0000", 2021-05-12T12:10:58.813 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:10:58.813 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.813 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.813 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:10:58.813 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:10:05.159518+0000", 2021-05-12T12:10:58.814 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:10:58.814 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:10:58.814 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:10:58.814 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:10:58.814 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:09:04.700046+0000", 2021-05-12T12:10:58.815 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:10:58.815 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.815 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.815 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap": [], 2021-05-12T12:10:58.815 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap_items": [], 2021-05-12T12:10:58.816 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_temp": [], 2021-05-12T12:10:58.816 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_temp": [], 2021-05-12T12:10:58.816 INFO:tasks.workunit.client.0.smithi191.stdout: "blocklist": {}, 2021-05-12T12:10:58.816 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profiles": { 2021-05-12T12:10:58.816 INFO:tasks.workunit.client.0.smithi191.stdout: "default": { 2021-05-12T12:10:58.817 INFO:tasks.workunit.client.0.smithi191.stdout: "k": "2", 2021-05-12T12:10:58.817 INFO:tasks.workunit.client.0.smithi191.stdout: "m": "2", 2021-05-12T12:10:58.817 INFO:tasks.workunit.client.0.smithi191.stdout: "plugin": "jerasure", 2021-05-12T12:10:58.817 INFO:tasks.workunit.client.0.smithi191.stdout: "technique": "reed_sol_van" 2021-05-12T12:10:58.817 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.818 INFO:tasks.workunit.client.0.smithi191.stdout: "myprofile": { 2021-05-12T12:10:58.818 INFO:tasks.workunit.client.0.smithi191.stdout: "crush-device-class": "", 2021-05-12T12:10:58.818 INFO:tasks.workunit.client.0.smithi191.stdout: "crush-failure-domain": "osd", 2021-05-12T12:10:58.818 INFO:tasks.workunit.client.0.smithi191.stdout: "crush-root": "default", 2021-05-12T12:10:58.818 INFO:tasks.workunit.client.0.smithi191.stdout: "jerasure-per-chunk-alignment": "false", 2021-05-12T12:10:58.819 INFO:tasks.workunit.client.0.smithi191.stdout: "k": "2", 2021-05-12T12:10:58.819 INFO:tasks.workunit.client.0.smithi191.stdout: "m": "1", 2021-05-12T12:10:58.819 INFO:tasks.workunit.client.0.smithi191.stdout: "plugin": "jerasure", 2021-05-12T12:10:58.819 INFO:tasks.workunit.client.0.smithi191.stdout: "technique": "reed_sol_van", 2021-05-12T12:10:58.819 INFO:tasks.workunit.client.0.smithi191.stdout: "w": "8" 2021-05-12T12:10:58.820 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.820 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.820 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps_queue": [], 2021-05-12T12:10:58.820 INFO:tasks.workunit.client.0.smithi191.stdout: "new_removed_snaps": [], 2021-05-12T12:10:58.821 INFO:tasks.workunit.client.0.smithi191.stdout: "new_purged_snaps": [], 2021-05-12T12:10:58.821 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_node_flags": {}, 2021-05-12T12:10:58.821 INFO:tasks.workunit.client.0.smithi191.stdout: "device_class_flags": {}, 2021-05-12T12:10:58.821 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": { 2021-05-12T12:10:58.821 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_enabled": false, 2021-05-12T12:10:58.822 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_bucket_count": 0, 2021-05-12T12:10:58.822 INFO:tasks.workunit.client.0.smithi191.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:10:58.822 INFO:tasks.workunit.client.0.smithi191.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:10:58.822 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_bucket": 0 2021-05-12T12:10:58.822 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.823 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.823 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_metadata": [ 2021-05-12T12:10:58.823 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.823 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:10:58.823 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:10:58.824 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6804/81623,v1:127.0.0.1:6805/81623]", 2021-05-12T12:10:58.824 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:10:58.824 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:10:58.824 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:10:58.824 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:10:58.825 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:10:58.825 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:10:58.825 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:10:58.825 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:10:58.826 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:10:58.826 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:10:58.826 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:10:58.826 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:10:58.826 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:10:58.827 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:10:58.827 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:10:58.827 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:10:58.827 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:10:58.827 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:10:58.827 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:10:58.828 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:10:58.828 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:10:58.828 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:10:58.828 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:10:58.828 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:10:58.829 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:10:58.829 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6802/81623,v1:127.0.0.1:6803/81623]", 2021-05-12T12:10:58.829 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:10:58.829 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/81623,v1:127.0.0.1:6809/81623]", 2021-05-12T12:10:58.829 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/81623,v1:127.0.0.1:6807/81623]", 2021-05-12T12:10:58.830 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:10:58.830 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:10:58.830 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:10:58.830 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:10:58.831 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:10:58.831 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:10:58.831 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:10:58.831 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:10:58.831 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:10:58.832 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:10:58.832 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:10:58.832 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:10:58.832 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:10:58.832 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.833 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.833 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:10:58.833 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:10:58.833 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/75793,v1:127.0.0.1:6813/75793]", 2021-05-12T12:10:58.833 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:10:58.834 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:10:58.834 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:10:58.834 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:10:58.834 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:10:58.835 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:10:58.835 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:10:58.835 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:10:58.835 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:10:58.835 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:10:58.836 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:10:58.836 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:10:58.836 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:10:58.836 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:10:58.837 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:10:58.837 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:10:58.837 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:10:58.837 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:10:58.837 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:10:58.838 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:10:58.838 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:10:58.838 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:10:58.838 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:10:58.838 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:10:58.839 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:10:58.839 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/75793,v1:127.0.0.1:6811/75793]", 2021-05-12T12:10:58.839 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:10:58.839 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/75793,v1:127.0.0.1:6817/75793]", 2021-05-12T12:10:58.839 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/75793,v1:127.0.0.1:6815/75793]", 2021-05-12T12:10:58.840 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:10:58.840 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:10:58.840 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:10:58.840 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:10:58.840 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:10:58.841 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:10:58.841 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:10:58.841 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:10:58.841 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:10:58.841 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:10:58.842 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:10:58.842 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:10:58.842 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:10:58.842 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.843 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.843 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:10:58.843 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:10:58.843 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/82206,v1:127.0.0.1:6813/82206]", 2021-05-12T12:10:58.843 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:10:58.843 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:10:58.844 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:10:58.844 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:10:58.844 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:10:58.844 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:10:58.844 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:10:58.845 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:10:58.845 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:10:58.845 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:10:58.845 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:10:58.845 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:10:58.846 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:10:58.846 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:10:58.846 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:10:58.846 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:10:58.847 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:10:58.847 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:10:58.847 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:10:58.847 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:10:58.847 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:10:58.848 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:10:58.848 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:10:58.848 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:10:58.848 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:10:58.848 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/82206,v1:127.0.0.1:6811/82206]", 2021-05-12T12:10:58.849 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:10:58.849 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/82206,v1:127.0.0.1:6817/82206]", 2021-05-12T12:10:58.849 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/82206,v1:127.0.0.1:6815/82206]", 2021-05-12T12:10:58.849 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:10:58.850 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:10:58.850 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:10:58.850 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:10:58.850 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:10:58.850 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:10:58.851 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:10:58.851 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:10:58.851 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:10:58.851 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:10:58.851 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:10:58.852 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:10:58.852 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:10:58.852 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.852 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.853 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_clean_epochs": { 2021-05-12T12:10:58.853 INFO:tasks.workunit.client.0.smithi191.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:10:58.853 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": { 2021-05-12T12:10:58.853 INFO:tasks.workunit.client.0.smithi191.stdout: "per_pool": [] 2021-05-12T12:10:58.853 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.854 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_epochs": [ 2021-05-12T12:10:58.854 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.854 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:10:58.854 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 35 2021-05-12T12:10:58.855 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.855 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.855 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:10:58.855 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 10 2021-05-12T12:10:58.855 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.856 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.856 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:10:58.856 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 37 2021-05-12T12:10:58.856 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.856 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.857 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.857 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_first_committed": 1, 2021-05-12T12:10:58.857 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_last_committed": 40, 2021-05-12T12:10:58.857 INFO:tasks.workunit.client.0.smithi191.stdout: "crushmap": { 2021-05-12T12:10:58.858 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": [ 2021-05-12T12:10:58.858 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.858 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:10:58.858 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.0", 2021-05-12T12:10:58.859 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:10:58.859 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.859 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.859 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:10:58.859 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.1", 2021-05-12T12:10:58.860 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:10:58.860 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.860 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.860 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:10:58.860 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.2", 2021-05-12T12:10:58.861 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:10:58.861 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.861 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.861 INFO:tasks.workunit.client.0.smithi191.stdout: "types": [ 2021-05-12T12:10:58.862 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.862 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 0, 2021-05-12T12:10:58.862 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd" 2021-05-12T12:10:58.862 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.862 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.863 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:10:58.863 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "host" 2021-05-12T12:10:58.863 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.863 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.863 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 2, 2021-05-12T12:10:58.864 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "chassis" 2021-05-12T12:10:58.864 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.864 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.864 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 3, 2021-05-12T12:10:58.864 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "rack" 2021-05-12T12:10:58.865 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.865 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.865 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 4, 2021-05-12T12:10:58.865 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "row" 2021-05-12T12:10:58.865 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.866 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.866 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 5, 2021-05-12T12:10:58.866 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pdu" 2021-05-12T12:10:58.866 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.866 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.867 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 6, 2021-05-12T12:10:58.867 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pod" 2021-05-12T12:10:58.867 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.867 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.867 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 7, 2021-05-12T12:10:58.868 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "room" 2021-05-12T12:10:58.868 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.868 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.868 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 8, 2021-05-12T12:10:58.869 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "datacenter" 2021-05-12T12:10:58.869 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.869 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.869 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 9, 2021-05-12T12:10:58.869 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "zone" 2021-05-12T12:10:58.870 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.870 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.870 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 10, 2021-05-12T12:10:58.870 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "region" 2021-05-12T12:10:58.871 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.871 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.871 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:10:58.871 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "root" 2021-05-12T12:10:58.871 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.872 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.872 INFO:tasks.workunit.client.0.smithi191.stdout: "buckets": [ 2021-05-12T12:10:58.873 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.873 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -1, 2021-05-12T12:10:58.873 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default", 2021-05-12T12:10:58.873 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:10:58.874 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:10:58.874 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:10:58.874 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:10:58.874 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:10:58.874 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:10:58.875 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.875 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:10:58.875 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:10:58.875 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:10:58.875 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.876 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.876 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.876 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.876 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -2, 2021-05-12T12:10:58.876 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default~hdd", 2021-05-12T12:10:58.877 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:10:58.877 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:10:58.877 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:10:58.877 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:10:58.877 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:10:58.878 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:10:58.878 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.878 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:10:58.878 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:10:58.878 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:10:58.879 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.879 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.879 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.879 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.879 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:10:58.880 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191", 2021-05-12T12:10:58.880 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:10:58.880 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:10:58.880 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:10:58.880 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:10:58.881 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:10:58.881 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:10:58.881 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.881 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:10:58.881 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:10:58.882 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:10:58.882 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.882 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.882 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:10:58.883 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:10:58.883 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:10:58.883 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.883 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.883 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:10:58.884 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:10:58.884 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:10:58.884 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.884 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.884 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.885 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.885 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:10:58.885 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191~hdd", 2021-05-12T12:10:58.885 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:10:58.885 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:10:58.886 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:10:58.886 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:10:58.886 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:10:58.886 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:10:58.886 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.887 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:10:58.887 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:10:58.887 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:10:58.887 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.887 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.888 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:10:58.888 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:10:58.888 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:10:58.888 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.888 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.889 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:10:58.889 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:10:58.889 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:10:58.889 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.889 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.890 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.890 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.890 INFO:tasks.workunit.client.0.smithi191.stdout: "rules": [ 2021-05-12T12:10:58.890 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.891 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 0, 2021-05-12T12:10:58.891 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "replicated_rule", 2021-05-12T12:10:58.891 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 0, 2021-05-12T12:10:58.891 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:10:58.892 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 1, 2021-05-12T12:10:58.892 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 10, 2021-05-12T12:10:58.892 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2021-05-12T12:10:58.892 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.892 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2021-05-12T12:10:58.893 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2021-05-12T12:10:58.893 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2021-05-12T12:10:58.893 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.893 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.893 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_firstn", 2021-05-12T12:10:58.894 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2021-05-12T12:10:58.894 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2021-05-12T12:10:58.894 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.894 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.894 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2021-05-12T12:10:58.895 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.895 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.895 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.895 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.895 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 1, 2021-05-12T12:10:58.896 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "test", 2021-05-12T12:10:58.896 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 1, 2021-05-12T12:10:58.896 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 3, 2021-05-12T12:10:58.896 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 3, 2021-05-12T12:10:58.896 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 3, 2021-05-12T12:10:58.897 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2021-05-12T12:10:58.897 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.897 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "set_chooseleaf_tries", 2021-05-12T12:10:58.897 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 5 2021-05-12T12:10:58.897 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.898 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.898 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "set_choose_tries", 2021-05-12T12:10:58.898 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 100 2021-05-12T12:10:58.898 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.898 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.899 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2021-05-12T12:10:58.899 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2021-05-12T12:10:58.899 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2021-05-12T12:10:58.899 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.899 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.900 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_indep", 2021-05-12T12:10:58.900 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2021-05-12T12:10:58.900 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2021-05-12T12:10:58.900 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.901 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.901 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2021-05-12T12:10:58.901 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.901 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:10:58.901 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.902 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.902 INFO:tasks.workunit.client.0.smithi191.stdout: "tunables": { 2021-05-12T12:10:58.902 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_tries": 0, 2021-05-12T12:10:58.902 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:10:58.902 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_total_tries": 50, 2021-05-12T12:10:58.903 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:10:58.903 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:10:58.903 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_stable": 1, 2021-05-12T12:10:58.903 INFO:tasks.workunit.client.0.smithi191.stdout: "straw_calc_version": 1, 2021-05-12T12:10:58.903 INFO:tasks.workunit.client.0.smithi191.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:10:58.904 INFO:tasks.workunit.client.0.smithi191.stdout: "profile": "jewel", 2021-05-12T12:10:58.904 INFO:tasks.workunit.client.0.smithi191.stdout: "optimal_tunables": 1, 2021-05-12T12:10:58.904 INFO:tasks.workunit.client.0.smithi191.stdout: "legacy_tunables": 0, 2021-05-12T12:10:58.904 INFO:tasks.workunit.client.0.smithi191.stdout: "minimum_required_version": "jewel", 2021-05-12T12:10:58.904 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables": 1, 2021-05-12T12:10:58.905 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables2": 1, 2021-05-12T12:10:58.905 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v2_rules": 1, 2021-05-12T12:10:58.905 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables3": 1, 2021-05-12T12:10:58.905 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v3_rules": 0, 2021-05-12T12:10:58.905 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v4_buckets": 1, 2021-05-12T12:10:58.906 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables5": 1, 2021-05-12T12:10:58.906 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v5_rules": 0 2021-05-12T12:10:58.906 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.906 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_args": {} 2021-05-12T12:10:58.906 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.907 INFO:tasks.workunit.client.0.smithi191.stdout: "fsmap": { 2021-05-12T12:10:58.907 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:10:58.907 INFO:tasks.workunit.client.0.smithi191.stdout: "default_fscid": -1, 2021-05-12T12:10:58.907 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": { 2021-05-12T12:10:58.907 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": {}, 2021-05-12T12:10:58.908 INFO:tasks.workunit.client.0.smithi191.stdout: "ro_compat": {}, 2021-05-12T12:10:58.908 INFO:tasks.workunit.client.0.smithi191.stdout: "incompat": { 2021-05-12T12:10:58.908 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_1": "base v0.20", 2021-05-12T12:10:58.908 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:10:58.908 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:10:58.909 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:10:58.909 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:10:58.909 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:10:58.909 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_8": "no anchor table", 2021-05-12T12:10:58.910 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_9": "file layout v2", 2021-05-12T12:10:58.910 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:10:58.910 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.910 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.910 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_flags": { 2021-05-12T12:10:58.911 INFO:tasks.workunit.client.0.smithi191.stdout: "enable_multiple": true, 2021-05-12T12:10:58.911 INFO:tasks.workunit.client.0.smithi191.stdout: "ever_enabled_multiple": true 2021-05-12T12:10:58.911 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.911 INFO:tasks.workunit.client.0.smithi191.stdout: "standbys": [], 2021-05-12T12:10:58.911 INFO:tasks.workunit.client.0.smithi191.stdout: "filesystems": [] 2021-05-12T12:10:58.912 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.912 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:10:58.912 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:10:58.912 INFO:tasks.workunit.client.0.smithi191.stdout: "auth": { 2021-05-12T12:10:58.912 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:10:58.913 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 5, 2021-05-12T12:10:58.913 INFO:tasks.workunit.client.0.smithi191.stdout: "num_secrets": 3 2021-05-12T12:10:58.913 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.913 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:10:58.913 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_active": 1, 2021-05-12T12:10:58.914 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_unknown": 0, 2021-05-12T12:10:58.914 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osd": 3, 2021-05-12T12:10:58.914 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_sum": { 2021-05-12T12:10:58.914 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:10:58.915 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48430, 2021-05-12T12:10:58.915 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 101, 2021-05-12T12:10:58.915 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 1, 2021-05-12T12:10:58.915 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 303, 2021-05-12T12:10:58.915 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:10:58.916 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:10:58.916 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 101, 2021-05-12T12:10:58.916 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:10:58.916 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:10:58.916 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 101, 2021-05-12T12:10:58.917 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:10:58.917 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:10:58.917 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:10:58.917 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 101, 2021-05-12T12:10:58.918 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 101, 2021-05-12T12:10:58.918 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:10:58.918 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:10:58.918 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:10:58.918 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:10:58.919 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:10:58.919 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:10:58.919 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:10:58.919 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:10:58.919 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:10:58.920 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:10:58.920 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:10:58.920 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:10:58.920 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:10:58.920 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:10:58.921 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:10:58.921 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:10:58.921 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:10:58.921 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:10:58.921 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:10:58.922 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:10:58.922 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:10:58.922 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:10:58.922 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:10:58.923 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:10:58.923 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:10:58.923 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.923 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:10:58.923 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:10:58.923 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:10:58.924 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:10:58.924 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 0, 2021-05-12T12:10:58.924 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 0, 2021-05-12T12:10:58.924 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:10:58.924 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:10:58.925 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:10:58.925 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:10:58.925 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:10:58.925 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.925 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 18, 2021-05-12T12:10:58.926 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 18, 2021-05-12T12:10:58.926 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 3, 2021-05-12T12:10:58.926 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 3, 2021-05-12T12:10:58.926 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 0 2021-05-12T12:10:58.927 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.927 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum": { 2021-05-12T12:10:58.927 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:10:58.927 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:10:58.927 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:10:58.927 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:10:58.928 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:10:58.928 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:10:58.928 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:10:58.928 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 18452, 2021-05-12T12:10:58.928 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2100, 2021-05-12T12:10:58.929 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:10:58.929 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16317, 2021-05-12T12:10:58.929 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314554348, 2021-05-12T12:10:58.929 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:10:58.929 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:10:58.930 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322103652352, 2021-05-12T12:10:58.930 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:10:58.930 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2150400, 2021-05-12T12:10:58.930 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 1526125, 2021-05-12T12:10:58.930 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:10:58.931 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:10:58.931 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:10:58.931 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:10:58.931 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 16709520 2021-05-12T12:10:58.931 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.932 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:10:58.932 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:10:58.932 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:10:58.932 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:10:58.932 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:10:58.933 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:10:58.933 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:10:58.933 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.933 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:10:58.933 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 0, 2021-05-12T12:10:58.934 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 0, 2021-05-12T12:10:58.934 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 0, 2021-05-12T12:10:58.934 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 0 2021-05-12T12:10:58.934 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.934 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:10:58.935 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:10:58.935 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.935 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum_by_class": { 2021-05-12T12:10:58.935 INFO:tasks.workunit.client.0.smithi191.stdout: "hdd": { 2021-05-12T12:10:58.936 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:10:58.936 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:10:58.936 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:10:58.936 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:10:58.936 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:10:58.937 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:10:58.937 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:10:58.937 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 18452, 2021-05-12T12:10:58.937 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2100, 2021-05-12T12:10:58.937 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 2, 2021-05-12T12:10:58.938 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16317, 2021-05-12T12:10:58.938 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314554348, 2021-05-12T12:10:58.938 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:10:58.938 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:10:58.938 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322103652352, 2021-05-12T12:10:58.939 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:10:58.939 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2150400, 2021-05-12T12:10:58.939 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 1526125, 2021-05-12T12:10:58.939 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:10:58.939 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:10:58.940 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:10:58.940 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 2160, 2021-05-12T12:10:58.940 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 16709520 2021-05-12T12:10:58.940 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.940 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:10:58.941 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:10:58.941 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:10:58.941 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:10:58.941 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:10:58.941 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:10:58.942 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:10:58.942 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.942 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:10:58.942 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 0, 2021-05-12T12:10:58.942 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 0, 2021-05-12T12:10:58.943 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 0, 2021-05-12T12:10:58.943 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 0 2021-05-12T12:10:58.943 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.943 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:10:58.943 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:10:58.944 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.944 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.944 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_stats": [ 2021-05-12T12:10:58.944 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.944 INFO:tasks.workunit.client.0.smithi191.stdout: "poolid": 1, 2021-05-12T12:10:58.945 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:10:58.945 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:10:58.945 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48430, 2021-05-12T12:10:58.945 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 101, 2021-05-12T12:10:58.945 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 1, 2021-05-12T12:10:58.946 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 303, 2021-05-12T12:10:58.946 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:10:58.946 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:10:58.946 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 101, 2021-05-12T12:10:58.946 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:10:58.947 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:10:58.947 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 101, 2021-05-12T12:10:58.947 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:10:58.947 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:10:58.947 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:10:58.948 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 101, 2021-05-12T12:10:58.948 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 101, 2021-05-12T12:10:58.948 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:10:58.948 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:10:58.948 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:10:58.949 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:10:58.949 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:10:58.949 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:10:58.949 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:10:58.949 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:10:58.949 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:10:58.950 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:10:58.950 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:10:58.950 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:10:58.950 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:10:58.950 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:10:58.951 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:10:58.951 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:10:58.951 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:10:58.951 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:10:58.952 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:10:58.952 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:10:58.952 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:10:58.952 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:10:58.952 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:10:58.952 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:10:58.953 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:10:58.953 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.953 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:10:58.953 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:10:58.953 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:10:58.954 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:10:58.954 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 1241088, 2021-05-12T12:10:58.954 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 1241088, 2021-05-12T12:10:58.954 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:10:58.955 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:10:58.955 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:10:58.955 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:10:58.955 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:10:58.955 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.956 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 18, 2021-05-12T12:10:58.956 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 18, 2021-05-12T12:10:58.956 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 3, 2021-05-12T12:10:58.956 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 3, 2021-05-12T12:10:58.956 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 3 2021-05-12T12:10:58.957 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.957 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.957 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_stats": [ 2021-05-12T12:10:58.957 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.958 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:10:58.958 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 150323855372 2021-05-12T12:10:58.958 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.958 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.959 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:10:58.959 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 42949672982 2021-05-12T12:10:58.959 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.959 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.960 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:10:58.960 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 158913789963 2021-05-12T12:10:58.960 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.960 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.960 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_state": [ 2021-05-12T12:10:58.961 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.961 INFO:tasks.workunit.client.0.smithi191.stdout: "state": "active+undersized+degraded", 2021-05-12T12:10:58.961 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 1 2021-05-12T12:10:58.961 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.962 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.962 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_osd": [ 2021-05-12T12:10:58.962 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.962 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:10:58.962 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:10:58.963 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:10:58.963 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:10:58.963 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.963 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.964 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2147483647, 2021-05-12T12:10:58.964 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:10:58.964 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:10:58.964 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:10:58.964 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.965 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.965 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:10:58.965 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:10:58.965 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 0, 2021-05-12T12:10:58.965 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:10:58.966 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.966 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.966 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:10:58.966 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 1, 2021-05-12T12:10:58.967 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:10:58.967 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:10:58.967 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.967 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.967 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": [ 2021-05-12T12:10:58.968 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:10:58.968 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:10:58.968 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": {} 2021-05-12T12:10:58.968 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.968 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:10:58.968 INFO:tasks.workunit.client.0.smithi191.stdout: "servicemap": { 2021-05-12T12:10:58.969 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 5, 2021-05-12T12:10:58.969 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:10:06.651673+0000", 2021-05-12T12:10:58.969 INFO:tasks.workunit.client.0.smithi191.stdout: "services": {} 2021-05-12T12:10:58.969 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:10:58.970 INFO:tasks.workunit.client.0.smithi191.stdout: "paxos": { 2021-05-12T12:10:58.970 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:10:58.970 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 414, 2021-05-12T12:10:58.970 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pn": 0, 2021-05-12T12:10:58.971 INFO:tasks.workunit.client.0.smithi191.stdout: "accepted_pn": 0 2021-05-12T12:10:58.971 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:10:58.971 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:10:58.973 INFO:tasks.workunit.client.0.smithi191.stderr:report 4085504885 2021-05-12T12:10:58.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:10:58.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:58.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2021-05-12T12:10:58.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:10:58.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:10:58.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:10:58.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:10:58.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:10:58.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:10:58.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2021-05-12T12:10:58.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:10:58.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:10:58.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=0 2021-05-12T12:10:58.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:10:58.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2021-05-12T12:10:58.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2021-05-12T12:10:59.778 INFO:tasks.workunit.client.0.smithi191.stderr:PG '1.0' not found 2021-05-12T12:11:00.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: _objectstore_tool_nodown: return 1 2021-05-12T12:11:00.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-05-12T12:11:00.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:11:00.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:11:00.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:11:00.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:11:00.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:11:00.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:11:00.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:11:00.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:11:00.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:11:00.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:11:00.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:11:00.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:11:00.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:11:00.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:11:00.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:11:00.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:11:00.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:11:00.302 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:11:00.302 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:00.302 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:11:00.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:11:00.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:11:00.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:11:00.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:11:00.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:11:00.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:11:00.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:11:00.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:00.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:11:00.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:11:00.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:11:00.305 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:11:00.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:11:00.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:00.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:11:00.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:11:00.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:11:00.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:11:00.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:11:00.329 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:00.326+0000 7fc6721290c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:00.331 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:00.328+0000 7fc6721290c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:00.333 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:00.330+0000 7fc6721290c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:00.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:11:00.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:11:00.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:11:00.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:11:00.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:11:00.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:11:00.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:11:00.676 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:11:00.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:11:00.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:11:00.853 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:00.851+0000 7fc6721290c0 -1 Falling back to public interface 2021-05-12T12:11:01.045 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 35 up_thru 39 down_at 29 last_clean_interval [5,27) [v2:127.0.0.1:6802/81623,v1:127.0.0.1:6803/81623] [v2:127.0.0.1:6804/81623,v1:127.0.0.1:6805/81623] exists,up 3164e607-150e-4cc6-bf11-92a6263c5c19 2021-05-12T12:11:01.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:11:01.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:11:01.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:11:01.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:11:01.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2021-05-12T12:11:01.048 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:11:01.048 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:11:01.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:11:01.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:11:01.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:11:01.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:11:01.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2021-05-12T12:11:01.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:11:01.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:11:01.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=2 2021-05-12T12:11:01.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:11:01.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2021-05-12T12:11:01.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2021-05-12T12:11:02.041 INFO:tasks.workunit.client.0.smithi191.stderr:PG '1.0' not found 2021-05-12T12:11:02.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: _objectstore_tool_nodown: return 1 2021-05-12T12:11:02.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-05-12T12:11:02.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:11:02.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:11:02.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:11:02.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:11:02.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:11:02.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:11:02.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:11:02.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:11:02.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:11:02.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:11:02.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:11:02.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:11:02.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:11:02.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:11:02.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:11:02.569 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:11:02.569 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:11:02.569 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:11:02.569 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:02.570 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:11:02.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:11:02.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:11:02.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:11:02.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:11:02.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:11:02.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:11:02.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:11:02.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:02.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:11:02.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:11:02.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:11:02.572 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:11:02.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:11:02.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:02.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:11:02.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:11:02.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:11:02.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:11:02.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:11:02.594 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:02.592+0000 7f4dd9ac10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:02.596 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:02.594+0000 7f4dd9ac10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:02.598 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:02.595+0000 7f4dd9ac10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:02.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:11:02.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:11:02.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:11:02.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:11:02.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:11:02.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:11:02.943 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:11:02.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:11:02.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:11:02.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:11:03.114 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:03.112+0000 7f4dd9ac10c0 -1 Falling back to public interface 2021-05-12T12:11:03.253 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:03.250+0000 7fc6721290c0 -1 osd.0 40 log_to_monitors {default=true} 2021-05-12T12:11:03.331 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 31 last_clean_interval [14,27) [v2:127.0.0.1:6810/82206,v1:127.0.0.1:6811/82206] [v2:127.0.0.1:6812/82206,v1:127.0.0.1:6813/82206] exists,up d6a6ae15-1b91-42f9-8ba3-5cd4469c3918 2021-05-12T12:11:03.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:11:03.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:11:03.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:11:03.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2021-05-12T12:11:03.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:11:03.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:11:03.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:11:03.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:11:03.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:11:03.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2021-05-12T12:11:05.062 INFO:tasks.workunit.client.0.smithi191.stderr:PG '1.0' not found 2021-05-12T12:11:05.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: _objectstore_tool_nodown: return 1 2021-05-12T12:11:05.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:11:05.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2021-05-12T12:11:05.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:11:05.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:11:05.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:11:05.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:11:05.335 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:11:05.335 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:11:05.335 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:11:05.335 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:11:05.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:11:05.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:11:05.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:11:05.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:11:05.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:11:05.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:11:05.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:11:05.474 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:05.471+0000 7f4dd9ac10c0 -1 osd.2 40 log_to_monitors {default=true} 2021-05-12T12:11:05.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:11:05.796 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:11:05.796 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:11:05.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:11:05.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:11:05.797 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:11:05.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=184683593731 2021-05-12T12:11:05.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 184683593731 2021-05-12T12:11:05.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593731' 2021-05-12T12:11:05.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:11:05.948 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:11:06.071 INFO:tasks.workunit.client.0.smithi191.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:11:06.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:11:06.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:11:06.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:11:06.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:11:06.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:11:06.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=201863462915 2021-05-12T12:11:06.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 201863462915 2021-05-12T12:11:06.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593731 2-201863462915' 2021-05-12T12:11:06.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:11:06.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-184683593731 2021-05-12T12:11:06.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:11:06.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:11:06.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-184683593731 2021-05-12T12:11:06.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:11:06.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=184683593731 2021-05-12T12:11:06.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 184683593731' 2021-05-12T12:11:06.237 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 184683593731 2021-05-12T12:11:06.237 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:11:06.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 1 -lt 184683593731 2021-05-12T12:11:06.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:11:07.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:11:07.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:11:08.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593731 -lt 184683593731 2021-05-12T12:11:08.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:11:08.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-201863462915 2021-05-12T12:11:08.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:11:08.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:11:08.031 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-201863462915 2021-05-12T12:11:08.031 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:11:08.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=201863462915 2021-05-12T12:11:08.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2021-05-12T12:11:08.032 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 201863462915 2021-05-12T12:11:08.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:11:08.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 201863462915 -lt 201863462915 2021-05-12T12:11:08.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:11:08.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:08.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:08.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:11:08.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:08.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:08.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:08.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:08.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:08.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:08.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:09.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:09.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:09.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:09.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:09.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:09.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:11:09.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:11:09.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:11:09.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:09.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:11:09.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:09.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:09.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:09.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:09.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:09.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:09.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:09.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:10.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:10.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:10.288 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:10.288 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:10.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:10.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:10.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:10.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:10.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:10.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:10.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:10.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:10.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:11.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:11.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:11.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:11:11.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:11.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:11:11.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:11.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:11.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:11.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:11.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:11.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:11.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:11.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:11.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:11.840 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:11.840 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:11.841 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:12.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:12.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:12.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:12.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:12.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:12.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:12.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:12.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:12.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:12.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:12.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:12.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:11:12.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:12.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:11:13.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:13.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:13.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:13.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:13.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:13.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:13.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:13.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:13.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:13.569 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:13.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:13.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:14.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:14.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:14.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:14.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:14.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:14.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:14.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:14.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:14.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:14.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:14.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:14.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:11:14.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:14.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:11:15.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:15.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:15.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:15.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:15.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:15.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:15.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:15.339 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:15.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:15.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:15.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:15.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:16.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:16.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:16.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:16.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:16.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:16.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:16.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:16.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:16.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:16.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:16.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:16.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:11:16.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:16.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:11:18.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:18.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:18.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:18.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:18.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:18.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:18.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:18.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:18.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:18.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:18.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:18.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:19.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:19.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:19.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:19.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:19.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:19.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:19.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:19.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:19.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:19.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:19.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:19.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:11:19.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:19.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:11:22.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:22.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:22.818 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:22.818 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:22.818 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:22.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:22.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:22.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:23.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:23.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:23.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:23.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:23.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:23.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:23.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:23.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:23.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:23.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:23.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:23.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:23.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:24.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:24.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:24.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:11:24.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:24.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:11:30.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:30.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:30.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:30.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:30.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:30.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:30.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:30.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:30.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:30.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:30.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:30.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:31.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:31.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:31.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:31.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:31.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:31.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:31.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:31.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:31.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:31.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:31.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:31.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:11:31.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:31.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:11:39.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:39.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:39.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:39.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:39.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:39.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:39.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:39.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:39.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:39.548 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:39.549 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:39.549 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:40.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:40.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:40.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:40.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:40.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:40.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:40.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:40.029 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:40.029 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:40.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:40.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:40.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:11:40.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:11:40.999 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:11:40.999 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_fingerprint": "7bf9d246-45ca-4870-98ba-18a9f48e5f82", 2021-05-12T12:11:40.999 INFO:tasks.workunit.client.0.smithi191.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:41.000 INFO:tasks.workunit.client.0.smithi191.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:11:41.000 INFO:tasks.workunit.client.0.smithi191.stdout: "timestamp": "2021-05-12T12:11:40.995814+0000", 2021-05-12T12:11:41.000 INFO:tasks.workunit.client.0.smithi191.stdout: "tag": "", 2021-05-12T12:11:41.000 INFO:tasks.workunit.client.0.smithi191.stdout: "health": { 2021-05-12T12:11:41.001 INFO:tasks.workunit.client.0.smithi191.stdout: "status": "HEALTH_ERR", 2021-05-12T12:11:41.001 INFO:tasks.workunit.client.0.smithi191.stdout: "checks": { 2021-05-12T12:11:41.001 INFO:tasks.workunit.client.0.smithi191.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:11:41.001 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:11:41.002 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:11:41.002 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:11:41.002 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:11:41.002 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.002 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:11:41.003 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.003 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:11:41.003 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.003 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.004 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:11:41.004 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.004 INFO:tasks.workunit.client.0.smithi191.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:11:41.004 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:41.004 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:11:41.005 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:11:41.005 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 17 2021-05-12T12:11:41.005 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.005 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [], 2021-05-12T12:11:41.006 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:11:41.006 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.006 INFO:tasks.workunit.client.0.smithi191.stdout: "OSD_DOWN": { 2021-05-12T12:11:41.006 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:41.006 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:11:41.007 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 osds down", 2021-05-12T12:11:41.007 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:11:41.007 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.007 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:11:41.007 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.008 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "osd.1 (root=default,host=smithi191) is down" 2021-05-12T12:11:41.008 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.008 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.008 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:11:41.009 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.009 INFO:tasks.workunit.client.0.smithi191.stdout: "PG_DEGRADED": { 2021-05-12T12:11:41.009 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:41.009 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:11:41.009 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2021-05-12T12:11:41.010 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:11:41.010 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.010 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:11:41.010 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.011 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2021-05-12T12:11:41.011 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.011 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.011 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:11:41.011 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.012 INFO:tasks.workunit.client.0.smithi191.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:11:41.012 INFO:tasks.workunit.client.0.smithi191.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:41.012 INFO:tasks.workunit.client.0.smithi191.stdout: "summary": { 2021-05-12T12:11:41.012 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:11:41.012 INFO:tasks.workunit.client.0.smithi191.stdout: "count": 1 2021-05-12T12:11:41.013 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.013 INFO:tasks.workunit.client.0.smithi191.stdout: "detail": [ 2021-05-12T12:11:41.013 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.013 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:11:41.013 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.014 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.014 INFO:tasks.workunit.client.0.smithi191.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:11:41.014 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.014 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.015 INFO:tasks.workunit.client.0.smithi191.stdout: "muted": false 2021-05-12T12:11:41.015 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.015 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.015 INFO:tasks.workunit.client.0.smithi191.stdout: "mutes": [] 2021-05-12T12:11:41.015 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.016 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_first_committed": 1, 2021-05-12T12:11:41.016 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap_last_committed": 1, 2021-05-12T12:11:41.016 INFO:tasks.workunit.client.0.smithi191.stdout: "monmap": { 2021-05-12T12:11:41.016 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:11:41.016 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:11:41.017 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:08:34.279716Z", 2021-05-12T12:11:41.017 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:08:34.279716Z", 2021-05-12T12:11:41.017 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release": 17, 2021-05-12T12:11:41.017 INFO:tasks.workunit.client.0.smithi191.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:11:41.017 INFO:tasks.workunit.client.0.smithi191.stdout: "election_strategy": 1, 2021-05-12T12:11:41.018 INFO:tasks.workunit.client.0.smithi191.stdout: "disallowed_leaders: ": "", 2021-05-12T12:11:41.018 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": false, 2021-05-12T12:11:41.018 INFO:tasks.workunit.client.0.smithi191.stdout: "features": { 2021-05-12T12:11:41.018 INFO:tasks.workunit.client.0.smithi191.stdout: "persistent": [ 2021-05-12T12:11:41.018 INFO:tasks.workunit.client.0.smithi191.stdout: "kraken", 2021-05-12T12:11:41.019 INFO:tasks.workunit.client.0.smithi191.stdout: "luminous", 2021-05-12T12:11:41.019 INFO:tasks.workunit.client.0.smithi191.stdout: "mimic", 2021-05-12T12:11:41.019 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap-prune", 2021-05-12T12:11:41.019 INFO:tasks.workunit.client.0.smithi191.stdout: "nautilus", 2021-05-12T12:11:41.019 INFO:tasks.workunit.client.0.smithi191.stdout: "octopus", 2021-05-12T12:11:41.020 INFO:tasks.workunit.client.0.smithi191.stdout: "pacific", 2021-05-12T12:11:41.020 INFO:tasks.workunit.client.0.smithi191.stdout: "elector-pinging", 2021-05-12T12:11:41.020 INFO:tasks.workunit.client.0.smithi191.stdout: "quincy" 2021-05-12T12:11:41.020 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.020 INFO:tasks.workunit.client.0.smithi191.stdout: "optional": [] 2021-05-12T12:11:41.021 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.021 INFO:tasks.workunit.client.0.smithi191.stdout: "mons": [ 2021-05-12T12:11:41.021 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.021 INFO:tasks.workunit.client.0.smithi191.stdout: "rank": 0, 2021-05-12T12:11:41.022 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "a", 2021-05-12T12:11:41.022 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:11:41.022 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.022 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.022 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.023 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:11:41.023 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 0 2021-05-12T12:11:41.023 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.023 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.023 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.024 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:11:41.024 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:11:41.024 INFO:tasks.workunit.client.0.smithi191.stdout: "priority": 0, 2021-05-12T12:11:41.024 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 0, 2021-05-12T12:11:41.024 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_location": "{}" 2021-05-12T12:11:41.025 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.025 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.025 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.025 INFO:tasks.workunit.client.0.smithi191.stdout: "quorum": [ 2021-05-12T12:11:41.026 INFO:tasks.workunit.client.0.smithi191.stdout: 0 2021-05-12T12:11:41.026 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.026 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap": { 2021-05-12T12:11:41.026 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 48, 2021-05-12T12:11:41.026 INFO:tasks.workunit.client.0.smithi191.stdout: "fsid": "58a97f67-bf94-4196-a5b7-5319dbe39c36", 2021-05-12T12:11:41.027 INFO:tasks.workunit.client.0.smithi191.stdout: "created": "2021-05-12T12:08:35.207927+0000", 2021-05-12T12:11:41.027 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:11:06.115736+0000", 2021-05-12T12:11:41.027 INFO:tasks.workunit.client.0.smithi191.stdout: "last_up_change": "2021-05-12T12:11:05.855775+0000", 2021-05-12T12:11:41.027 INFO:tasks.workunit.client.0.smithi191.stdout: "last_in_change": "2021-05-12T12:10:17.579142+0000", 2021-05-12T12:11:41.027 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:11:41.028 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_num": 5800384, 2021-05-12T12:11:41.028 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_set": [ 2021-05-12T12:11:41.028 INFO:tasks.workunit.client.0.smithi191.stdout: "nodown", 2021-05-12T12:11:41.028 INFO:tasks.workunit.client.0.smithi191.stdout: "noin", 2021-05-12T12:11:41.029 INFO:tasks.workunit.client.0.smithi191.stdout: "noout", 2021-05-12T12:11:41.029 INFO:tasks.workunit.client.0.smithi191.stdout: "pglog_hardlimit", 2021-05-12T12:11:41.029 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snapdirs", 2021-05-12T12:11:41.029 INFO:tasks.workunit.client.0.smithi191.stdout: "recovery_deletes", 2021-05-12T12:11:41.029 INFO:tasks.workunit.client.0.smithi191.stdout: "sortbitwise" 2021-05-12T12:11:41.030 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.030 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_version": 12, 2021-05-12T12:11:41.030 INFO:tasks.workunit.client.0.smithi191.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:11:41.030 INFO:tasks.workunit.client.0.smithi191.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:11:41.030 INFO:tasks.workunit.client.0.smithi191.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:11:41.031 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_snapshot": "", 2021-05-12T12:11:41.031 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_max": 1, 2021-05-12T12:11:41.031 INFO:tasks.workunit.client.0.smithi191.stdout: "max_osd": 3, 2021-05-12T12:11:41.031 INFO:tasks.workunit.client.0.smithi191.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:11:41.032 INFO:tasks.workunit.client.0.smithi191.stdout: "min_compat_client": "jewel", 2021-05-12T12:11:41.032 INFO:tasks.workunit.client.0.smithi191.stdout: "require_osd_release": "quincy", 2021-05-12T12:11:41.032 INFO:tasks.workunit.client.0.smithi191.stdout: "pools": [ 2021-05-12T12:11:41.032 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.032 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:11:41.033 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_name": "test", 2021-05-12T12:11:41.033 INFO:tasks.workunit.client.0.smithi191.stdout: "create_time": "2021-05-12T12:09:07.671909+0000", 2021-05-12T12:11:41.033 INFO:tasks.workunit.client.0.smithi191.stdout: "flags": 16389, 2021-05-12T12:11:41.033 INFO:tasks.workunit.client.0.smithi191.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-05-12T12:11:41.033 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 3, 2021-05-12T12:11:41.034 INFO:tasks.workunit.client.0.smithi191.stdout: "size": 3, 2021-05-12T12:11:41.034 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 2, 2021-05-12T12:11:41.034 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_rule": 1, 2021-05-12T12:11:41.034 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:11:41.035 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:11:41.035 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:11:41.035 INFO:tasks.workunit.client.0.smithi191.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:11:41.035 INFO:tasks.workunit.client.0.smithi191.stdout: "object_hash": 2, 2021-05-12T12:11:41.035 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:11:41.036 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num": 1, 2021-05-12T12:11:41.036 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num": 1, 2021-05-12T12:11:41.036 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_placement_num_target": 1, 2021-05-12T12:11:41.036 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_target": 1, 2021-05-12T12:11:41.036 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_num_pending": 1, 2021-05-12T12:11:41.037 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pg_merge_meta": { 2021-05-12T12:11:41.037 INFO:tasks.workunit.client.0.smithi191.stdout: "source_pgid": "0.0", 2021-05-12T12:11:41.037 INFO:tasks.workunit.client.0.smithi191.stdout: "ready_epoch": 0, 2021-05-12T12:11:41.037 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_started": 0, 2021-05-12T12:11:41.038 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": 0, 2021-05-12T12:11:41.038 INFO:tasks.workunit.client.0.smithi191.stdout: "source_version": "0'0", 2021-05-12T12:11:41.038 INFO:tasks.workunit.client.0.smithi191.stdout: "target_version": "0'0" 2021-05-12T12:11:41.038 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.038 INFO:tasks.workunit.client.0.smithi191.stdout: "last_change": "28", 2021-05-12T12:11:41.039 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend": "0", 2021-05-12T12:11:41.039 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:11:41.039 INFO:tasks.workunit.client.0.smithi191.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:11:41.039 INFO:tasks.workunit.client.0.smithi191.stdout: "auid": 0, 2021-05-12T12:11:41.039 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_mode": "pool", 2021-05-12T12:11:41.040 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_seq": 1, 2021-05-12T12:11:41.040 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_epoch": 28, 2021-05-12T12:11:41.040 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_snaps": [ 2021-05-12T12:11:41.041 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.041 INFO:tasks.workunit.client.0.smithi191.stdout: "snapid": 1, 2021-05-12T12:11:41.041 INFO:tasks.workunit.client.0.smithi191.stdout: "stamp": "2021-05-12T12:09:53.148800+0000", 2021-05-12T12:11:41.041 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "snap1" 2021-05-12T12:11:41.041 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.042 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.042 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps": "[]", 2021-05-12T12:11:41.042 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_bytes": 0, 2021-05-12T12:11:41.042 INFO:tasks.workunit.client.0.smithi191.stdout: "quota_max_objects": 0, 2021-05-12T12:11:41.043 INFO:tasks.workunit.client.0.smithi191.stdout: "tiers": [], 2021-05-12T12:11:41.043 INFO:tasks.workunit.client.0.smithi191.stdout: "tier_of": -1, 2021-05-12T12:11:41.043 INFO:tasks.workunit.client.0.smithi191.stdout: "read_tier": -1, 2021-05-12T12:11:41.043 INFO:tasks.workunit.client.0.smithi191.stdout: "write_tier": -1, 2021-05-12T12:11:41.043 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_mode": "none", 2021-05-12T12:11:41.044 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_bytes": 0, 2021-05-12T12:11:41.044 INFO:tasks.workunit.client.0.smithi191.stdout: "target_max_objects": 0, 2021-05-12T12:11:41.044 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:11:41.044 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:11:41.044 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:11:41.045 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_flush_age": 0, 2021-05-12T12:11:41.045 INFO:tasks.workunit.client.0.smithi191.stdout: "cache_min_evict_age": 0, 2021-05-12T12:11:41.045 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profile": "myprofile", 2021-05-12T12:11:41.045 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_params": { 2021-05-12T12:11:41.046 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "none" 2021-05-12T12:11:41.046 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.046 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_period": 0, 2021-05-12T12:11:41.046 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_count": 0, 2021-05-12T12:11:41.046 INFO:tasks.workunit.client.0.smithi191.stdout: "use_gmt_hitset": true, 2021-05-12T12:11:41.047 INFO:tasks.workunit.client.0.smithi191.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:11:41.047 INFO:tasks.workunit.client.0.smithi191.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:11:41.047 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:11:41.047 INFO:tasks.workunit.client.0.smithi191.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:11:41.047 INFO:tasks.workunit.client.0.smithi191.stdout: "grade_table": [], 2021-05-12T12:11:41.048 INFO:tasks.workunit.client.0.smithi191.stdout: "stripe_width": 8192, 2021-05-12T12:11:41.048 INFO:tasks.workunit.client.0.smithi191.stdout: "expected_num_objects": 0, 2021-05-12T12:11:41.048 INFO:tasks.workunit.client.0.smithi191.stdout: "fast_read": false, 2021-05-12T12:11:41.048 INFO:tasks.workunit.client.0.smithi191.stdout: "options": {}, 2021-05-12T12:11:41.049 INFO:tasks.workunit.client.0.smithi191.stdout: "application_metadata": {} 2021-05-12T12:11:41.049 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.049 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.049 INFO:tasks.workunit.client.0.smithi191.stdout: "osds": [ 2021-05-12T12:11:41.049 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.050 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:11:41.050 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "3164e607-150e-4cc6-bf11-92a6263c5c19", 2021-05-12T12:11:41.050 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:11:41.050 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:11:41.050 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:11:41.051 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:11:41.051 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 35, 2021-05-12T12:11:41.051 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 40, 2021-05-12T12:11:41.051 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 43, 2021-05-12T12:11:41.051 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 47, 2021-05-12T12:11:41.052 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 42, 2021-05-12T12:11:41.052 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:11:41.052 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:11:41.052 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.052 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.053 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.053 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:11:41.053 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84260 2021-05-12T12:11:41.053 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.053 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.054 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.054 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:11:41.054 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84260 2021-05-12T12:11:41.054 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.055 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.055 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.055 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:11:41.055 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.055 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.056 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.056 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:11:41.056 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84260 2021-05-12T12:11:41.056 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.056 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.057 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.057 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:11:41.057 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84260 2021-05-12T12:11:41.057 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.058 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.058 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.058 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:11:41.058 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.058 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.059 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.059 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:11:41.059 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84260 2021-05-12T12:11:41.059 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.059 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.060 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.060 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:11:41.060 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84260 2021-05-12T12:11:41.060 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.060 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.061 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.061 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:11:41.061 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.061 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.062 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.062 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:11:41.062 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84260 2021-05-12T12:11:41.062 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.062 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.063 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.063 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:11:41.063 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84260 2021-05-12T12:11:41.063 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.063 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.064 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.064 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6803/84260", 2021-05-12T12:11:41.064 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6805/84260", 2021-05-12T12:11:41.064 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6809/84260", 2021-05-12T12:11:41.064 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6807/84260", 2021-05-12T12:11:41.065 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:11:41.065 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:11:41.065 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:11:41.065 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.065 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.066 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.066 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:11:41.066 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "4249e897-1cf7-49ad-8d28-f401f2373f5a", 2021-05-12T12:11:41.066 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 0, 2021-05-12T12:11:41.067 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:11:41.067 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:11:41.067 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:11:41.067 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 0, 2021-05-12T12:11:41.067 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 0, 2021-05-12T12:11:41.068 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 10, 2021-05-12T12:11:41.068 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 23, 2021-05-12T12:11:41.068 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 30, 2021-05-12T12:11:41.068 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:11:41.068 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:11:41.069 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.069 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.069 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.069 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:11:41.069 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:11:41.070 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.070 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.070 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.070 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:11:41.070 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:11:41.071 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.071 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.071 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.071 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:11:41.071 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.072 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.072 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.072 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:11:41.072 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:11:41.072 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.073 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.073 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.073 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:11:41.073 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:11:41.073 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.074 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.074 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.074 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:11:41.074 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.074 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.075 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.075 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:11:41.075 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:11:41.075 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.075 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.076 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.076 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:11:41.076 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:11:41.076 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.076 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.077 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.077 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:11:41.077 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.077 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.077 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.078 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:11:41.078 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:11:41.078 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.078 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.078 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.079 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:11:41.079 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 75793 2021-05-12T12:11:41.079 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.079 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.079 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.080 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/75793", 2021-05-12T12:11:41.080 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/75793", 2021-05-12T12:11:41.080 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/75793", 2021-05-12T12:11:41.080 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/75793", 2021-05-12T12:11:41.080 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:11:41.081 INFO:tasks.workunit.client.0.smithi191.stdout: "exists" 2021-05-12T12:11:41.081 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.081 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.081 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.081 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:11:41.081 INFO:tasks.workunit.client.0.smithi191.stdout: "uuid": "d6a6ae15-1b91-42f9-8ba3-5cd4469c3918", 2021-05-12T12:11:41.082 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 1, 2021-05-12T12:11:41.082 INFO:tasks.workunit.client.0.smithi191.stdout: "in": 1, 2021-05-12T12:11:41.082 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 1, 2021-05-12T12:11:41.082 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_affinity": 1, 2021-05-12T12:11:41.083 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_begin": 37, 2021-05-12T12:11:41.083 INFO:tasks.workunit.client.0.smithi191.stdout: "last_clean_end": 40, 2021-05-12T12:11:41.083 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 47, 2021-05-12T12:11:41.083 INFO:tasks.workunit.client.0.smithi191.stdout: "up_thru": 0, 2021-05-12T12:11:41.083 INFO:tasks.workunit.client.0.smithi191.stdout: "down_at": 46, 2021-05-12T12:11:41.084 INFO:tasks.workunit.client.0.smithi191.stdout: "lost_at": 0, 2021-05-12T12:11:41.084 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addrs": { 2021-05-12T12:11:41.084 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.084 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.084 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.085 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:11:41.085 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84943 2021-05-12T12:11:41.085 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.085 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.085 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.086 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:11:41.086 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84943 2021-05-12T12:11:41.086 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.086 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.086 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.087 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addrs": { 2021-05-12T12:11:41.087 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.087 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.087 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.087 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:11:41.088 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84943 2021-05-12T12:11:41.088 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.088 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.088 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.089 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:11:41.089 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84943 2021-05-12T12:11:41.089 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.089 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.089 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.090 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addrs": { 2021-05-12T12:11:41.090 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.090 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.090 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.090 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:11:41.091 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84943 2021-05-12T12:11:41.091 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.091 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.091 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.091 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:11:41.092 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84943 2021-05-12T12:11:41.092 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.092 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.092 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.093 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addrs": { 2021-05-12T12:11:41.093 INFO:tasks.workunit.client.0.smithi191.stdout: "addrvec": [ 2021-05-12T12:11:41.093 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.093 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v2", 2021-05-12T12:11:41.093 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:11:41.094 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84943 2021-05-12T12:11:41.094 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.094 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.094 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "v1", 2021-05-12T12:11:41.095 INFO:tasks.workunit.client.0.smithi191.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:11:41.095 INFO:tasks.workunit.client.0.smithi191.stdout: "nonce": 84943 2021-05-12T12:11:41.095 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.095 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.095 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.096 INFO:tasks.workunit.client.0.smithi191.stdout: "public_addr": "127.0.0.1:6811/84943", 2021-05-12T12:11:41.096 INFO:tasks.workunit.client.0.smithi191.stdout: "cluster_addr": "127.0.0.1:6813/84943", 2021-05-12T12:11:41.096 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_back_addr": "127.0.0.1:6817/84943", 2021-05-12T12:11:41.096 INFO:tasks.workunit.client.0.smithi191.stdout: "heartbeat_front_addr": "127.0.0.1:6815/84943", 2021-05-12T12:11:41.096 INFO:tasks.workunit.client.0.smithi191.stdout: "state": [ 2021-05-12T12:11:41.097 INFO:tasks.workunit.client.0.smithi191.stdout: "exists", 2021-05-12T12:11:41.097 INFO:tasks.workunit.client.0.smithi191.stdout: "up" 2021-05-12T12:11:41.097 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.097 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.097 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.098 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_xinfo": [ 2021-05-12T12:11:41.098 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.098 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:11:41.098 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:11:03.572411+0000", 2021-05-12T12:11:41.098 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:11:41.099 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:11:41.099 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:11:41.099 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:11:41.099 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:08:45.451069+0000", 2021-05-12T12:11:41.099 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:11:41.100 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.100 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.100 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:11:41.100 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:10:04.641605+0000", 2021-05-12T12:11:41.101 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:11:41.101 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:11:41.101 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:11:41.101 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:11:41.101 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:08:54.873051+0000", 2021-05-12T12:11:41.102 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:11:41.102 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.102 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.102 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:11:41.103 INFO:tasks.workunit.client.0.smithi191.stdout: "down_stamp": "2021-05-12T12:11:05.671793+0000", 2021-05-12T12:11:41.103 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_probability": 0, 2021-05-12T12:11:41.103 INFO:tasks.workunit.client.0.smithi191.stdout: "laggy_interval": 0, 2021-05-12T12:11:41.103 INFO:tasks.workunit.client.0.smithi191.stdout: "features": 4540138303579357183, 2021-05-12T12:11:41.103 INFO:tasks.workunit.client.0.smithi191.stdout: "old_weight": 0, 2021-05-12T12:11:41.104 INFO:tasks.workunit.client.0.smithi191.stdout: "last_purged_snaps_scrub": "2021-05-12T12:09:04.700046+0000", 2021-05-12T12:11:41.104 INFO:tasks.workunit.client.0.smithi191.stdout: "dead_epoch": 0 2021-05-12T12:11:41.104 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.104 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.104 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap": [], 2021-05-12T12:11:41.105 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_upmap_items": [], 2021-05-12T12:11:41.105 INFO:tasks.workunit.client.0.smithi191.stdout: "pg_temp": [], 2021-05-12T12:11:41.105 INFO:tasks.workunit.client.0.smithi191.stdout: "primary_temp": [], 2021-05-12T12:11:41.105 INFO:tasks.workunit.client.0.smithi191.stdout: "blocklist": {}, 2021-05-12T12:11:41.105 INFO:tasks.workunit.client.0.smithi191.stdout: "erasure_code_profiles": { 2021-05-12T12:11:41.106 INFO:tasks.workunit.client.0.smithi191.stdout: "default": { 2021-05-12T12:11:41.106 INFO:tasks.workunit.client.0.smithi191.stdout: "k": "2", 2021-05-12T12:11:41.106 INFO:tasks.workunit.client.0.smithi191.stdout: "m": "2", 2021-05-12T12:11:41.106 INFO:tasks.workunit.client.0.smithi191.stdout: "plugin": "jerasure", 2021-05-12T12:11:41.107 INFO:tasks.workunit.client.0.smithi191.stdout: "technique": "reed_sol_van" 2021-05-12T12:11:41.107 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.107 INFO:tasks.workunit.client.0.smithi191.stdout: "myprofile": { 2021-05-12T12:11:41.107 INFO:tasks.workunit.client.0.smithi191.stdout: "crush-device-class": "", 2021-05-12T12:11:41.107 INFO:tasks.workunit.client.0.smithi191.stdout: "crush-failure-domain": "osd", 2021-05-12T12:11:41.108 INFO:tasks.workunit.client.0.smithi191.stdout: "crush-root": "default", 2021-05-12T12:11:41.108 INFO:tasks.workunit.client.0.smithi191.stdout: "jerasure-per-chunk-alignment": "false", 2021-05-12T12:11:41.108 INFO:tasks.workunit.client.0.smithi191.stdout: "k": "2", 2021-05-12T12:11:41.108 INFO:tasks.workunit.client.0.smithi191.stdout: "m": "1", 2021-05-12T12:11:41.108 INFO:tasks.workunit.client.0.smithi191.stdout: "plugin": "jerasure", 2021-05-12T12:11:41.109 INFO:tasks.workunit.client.0.smithi191.stdout: "technique": "reed_sol_van", 2021-05-12T12:11:41.109 INFO:tasks.workunit.client.0.smithi191.stdout: "w": "8" 2021-05-12T12:11:41.109 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.109 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.110 INFO:tasks.workunit.client.0.smithi191.stdout: "removed_snaps_queue": [], 2021-05-12T12:11:41.110 INFO:tasks.workunit.client.0.smithi191.stdout: "new_removed_snaps": [], 2021-05-12T12:11:41.110 INFO:tasks.workunit.client.0.smithi191.stdout: "new_purged_snaps": [], 2021-05-12T12:11:41.110 INFO:tasks.workunit.client.0.smithi191.stdout: "crush_node_flags": {}, 2021-05-12T12:11:41.110 INFO:tasks.workunit.client.0.smithi191.stdout: "device_class_flags": {}, 2021-05-12T12:11:41.111 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode": { 2021-05-12T12:11:41.111 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_enabled": false, 2021-05-12T12:11:41.111 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_bucket_count": 0, 2021-05-12T12:11:41.111 INFO:tasks.workunit.client.0.smithi191.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:11:41.111 INFO:tasks.workunit.client.0.smithi191.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:11:41.112 INFO:tasks.workunit.client.0.smithi191.stdout: "stretch_mode_bucket": 0 2021-05-12T12:11:41.112 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.112 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.112 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_metadata": [ 2021-05-12T12:11:41.112 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.113 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:11:41.113 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:11:41.113 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6804/84260,v1:127.0.0.1:6805/84260]", 2021-05-12T12:11:41.113 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:11:41.113 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:11:41.114 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:11:41.114 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:11:41.114 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:11:41.114 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:11:41.114 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:11:41.115 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:11:41.115 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:11:41.115 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:11:41.115 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:11:41.115 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:11:41.116 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:11:41.116 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:11:41.116 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:11:41.116 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:11:41.116 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:41.117 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:11:41.117 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:11:41.117 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:11:41.117 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:11:41.117 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:11:41.118 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:11:41.118 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:11:41.118 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:11:41.118 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6802/84260,v1:127.0.0.1:6803/84260]", 2021-05-12T12:11:41.119 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:11:41.119 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/84260,v1:127.0.0.1:6809/84260]", 2021-05-12T12:11:41.119 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/84260,v1:127.0.0.1:6807/84260]", 2021-05-12T12:11:41.119 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:11:41.119 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:11:41.120 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:11:41.120 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:11:41.120 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:11:41.120 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:11:41.120 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:11:41.121 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:11:41.121 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:11:41.121 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:11:41.121 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:11:41.121 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:11:41.122 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:11:41.122 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.122 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.122 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:11:41.123 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:11:41.123 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/75793,v1:127.0.0.1:6813/75793]", 2021-05-12T12:11:41.123 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:11:41.123 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:11:41.123 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:11:41.124 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:11:41.124 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:11:41.124 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:11:41.124 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:11:41.125 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:11:41.125 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:11:41.125 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:11:41.125 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:11:41.125 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:11:41.126 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:11:41.126 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:11:41.126 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:11:41.126 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:11:41.127 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:41.127 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:11:41.127 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:11:41.127 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:11:41.128 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:11:41.128 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:11:41.128 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:11:41.128 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:11:41.128 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:11:41.129 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/75793,v1:127.0.0.1:6811/75793]", 2021-05-12T12:11:41.129 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:11:41.129 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/75793,v1:127.0.0.1:6817/75793]", 2021-05-12T12:11:41.129 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/75793,v1:127.0.0.1:6815/75793]", 2021-05-12T12:11:41.129 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:11:41.130 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:11:41.130 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:11:41.130 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:11:41.130 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:11:41.130 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:11:41.131 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:11:41.131 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:11:41.131 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:11:41.131 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:11:41.131 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:11:41.132 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:11:41.132 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:11:41.132 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.132 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.133 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:11:41.133 INFO:tasks.workunit.client.0.smithi191.stdout: "arch": "x86_64", 2021-05-12T12:11:41.133 INFO:tasks.workunit.client.0.smithi191.stdout: "back_addr": "[v2:127.0.0.1:6812/84943,v1:127.0.0.1:6813/84943]", 2021-05-12T12:11:41.133 INFO:tasks.workunit.client.0.smithi191.stdout: "back_iface": "", 2021-05-12T12:11:41.133 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs": "1", 2021-05-12T12:11:41.134 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:11:41.134 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:11:41.134 INFO:tasks.workunit.client.0.smithi191.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:11:41.134 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:11:41.134 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:11:41.135 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:11:41.135 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:11:41.135 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:11:41.135 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:11:41.135 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:11:41.135 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:11:41.136 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:11:41.136 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_release": "quincy", 2021-05-12T12:11:41.136 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:11:41.136 INFO:tasks.workunit.client.0.smithi191.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:41.136 INFO:tasks.workunit.client.0.smithi191.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:11:41.137 INFO:tasks.workunit.client.0.smithi191.stdout: "default_device_class": "hdd", 2021-05-12T12:11:41.137 INFO:tasks.workunit.client.0.smithi191.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XMDN", 2021-05-12T12:11:41.137 INFO:tasks.workunit.client.0.smithi191.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:11:41.137 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": "sda", 2021-05-12T12:11:41.137 INFO:tasks.workunit.client.0.smithi191.stdout: "distro": "rhel", 2021-05-12T12:11:41.138 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_description": "Red Hat Enterprise Linux 8.3 (Ootpa)", 2021-05-12T12:11:41.138 INFO:tasks.workunit.client.0.smithi191.stdout: "distro_version": "8.3", 2021-05-12T12:11:41.138 INFO:tasks.workunit.client.0.smithi191.stdout: "front_addr": "[v2:127.0.0.1:6810/84943,v1:127.0.0.1:6811/84943]", 2021-05-12T12:11:41.138 INFO:tasks.workunit.client.0.smithi191.stdout: "front_iface": "", 2021-05-12T12:11:41.138 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/84943,v1:127.0.0.1:6817/84943]", 2021-05-12T12:11:41.139 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/84943,v1:127.0.0.1:6815/84943]", 2021-05-12T12:11:41.139 INFO:tasks.workunit.client.0.smithi191.stdout: "hostname": "smithi191", 2021-05-12T12:11:41.139 INFO:tasks.workunit.client.0.smithi191.stdout: "journal_rotational": "1", 2021-05-12T12:11:41.139 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_description": "#1 SMP Fri Oct 16 13:36:46 EDT 2020", 2021-05-12T12:11:41.140 INFO:tasks.workunit.client.0.smithi191.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-05-12T12:11:41.140 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_swap_kb": "0", 2021-05-12T12:11:41.140 INFO:tasks.workunit.client.0.smithi191.stdout: "mem_total_kb": "32610028", 2021-05-12T12:11:41.140 INFO:tasks.workunit.client.0.smithi191.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:11:41.140 INFO:tasks.workunit.client.0.smithi191.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:11:41.141 INFO:tasks.workunit.client.0.smithi191.stdout: "os": "Linux", 2021-05-12T12:11:41.141 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:11:41.141 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:11:41.141 INFO:tasks.workunit.client.0.smithi191.stdout: "osdspec_affinity": "", 2021-05-12T12:11:41.141 INFO:tasks.workunit.client.0.smithi191.stdout: "rotational": "1" 2021-05-12T12:11:41.142 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.142 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.142 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_clean_epochs": { 2021-05-12T12:11:41.142 INFO:tasks.workunit.client.0.smithi191.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:11:41.143 INFO:tasks.workunit.client.0.smithi191.stdout: "last_epoch_clean": { 2021-05-12T12:11:41.143 INFO:tasks.workunit.client.0.smithi191.stdout: "per_pool": [] 2021-05-12T12:11:41.143 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.143 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_epochs": [ 2021-05-12T12:11:41.143 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.144 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:11:41.144 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 43 2021-05-12T12:11:41.144 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.144 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.144 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:11:41.145 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 10 2021-05-12T12:11:41.145 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.145 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.145 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:11:41.146 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 48 2021-05-12T12:11:41.146 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.146 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.146 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.146 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_first_committed": 1, 2021-05-12T12:11:41.147 INFO:tasks.workunit.client.0.smithi191.stdout: "osdmap_last_committed": 48, 2021-05-12T12:11:41.147 INFO:tasks.workunit.client.0.smithi191.stdout: "crushmap": { 2021-05-12T12:11:41.147 INFO:tasks.workunit.client.0.smithi191.stdout: "devices": [ 2021-05-12T12:11:41.147 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.147 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:11:41.148 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.0", 2021-05-12T12:11:41.148 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:11:41.148 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.148 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.149 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:11:41.149 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.1", 2021-05-12T12:11:41.149 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:11:41.149 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.149 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.150 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:11:41.150 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd.2", 2021-05-12T12:11:41.150 INFO:tasks.workunit.client.0.smithi191.stdout: "class": "hdd" 2021-05-12T12:11:41.150 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.150 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.151 INFO:tasks.workunit.client.0.smithi191.stdout: "types": [ 2021-05-12T12:11:41.151 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.151 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 0, 2021-05-12T12:11:41.151 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "osd" 2021-05-12T12:11:41.151 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.152 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.152 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:11:41.152 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "host" 2021-05-12T12:11:41.152 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.153 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.153 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 2, 2021-05-12T12:11:41.153 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "chassis" 2021-05-12T12:11:41.153 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.153 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.154 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 3, 2021-05-12T12:11:41.154 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "rack" 2021-05-12T12:11:41.154 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.154 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.154 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 4, 2021-05-12T12:11:41.155 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "row" 2021-05-12T12:11:41.155 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.155 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.155 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 5, 2021-05-12T12:11:41.156 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pdu" 2021-05-12T12:11:41.156 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.156 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.156 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 6, 2021-05-12T12:11:41.156 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "pod" 2021-05-12T12:11:41.157 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.157 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.157 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 7, 2021-05-12T12:11:41.157 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "room" 2021-05-12T12:11:41.157 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.158 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.158 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 8, 2021-05-12T12:11:41.158 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "datacenter" 2021-05-12T12:11:41.158 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.158 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.159 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 9, 2021-05-12T12:11:41.159 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "zone" 2021-05-12T12:11:41.159 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.159 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.159 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 10, 2021-05-12T12:11:41.160 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "region" 2021-05-12T12:11:41.160 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.160 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.160 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:11:41.160 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "root" 2021-05-12T12:11:41.161 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.161 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.161 INFO:tasks.workunit.client.0.smithi191.stdout: "buckets": [ 2021-05-12T12:11:41.161 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.161 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -1, 2021-05-12T12:11:41.162 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default", 2021-05-12T12:11:41.162 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:11:41.162 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:11:41.162 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:11:41.163 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:11:41.163 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:11:41.163 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:11:41.163 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.163 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:11:41.164 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:11:41.164 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:11:41.164 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.164 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.164 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.164 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.165 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -2, 2021-05-12T12:11:41.165 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "default~hdd", 2021-05-12T12:11:41.165 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 11, 2021-05-12T12:11:41.165 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "root", 2021-05-12T12:11:41.166 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:11:41.166 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:11:41.166 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:11:41.166 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:11:41.166 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.167 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:11:41.167 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:11:41.167 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:11:41.167 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.167 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.168 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.168 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.168 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -3, 2021-05-12T12:11:41.168 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191", 2021-05-12T12:11:41.168 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:11:41.169 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:11:41.169 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:11:41.169 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:11:41.169 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:11:41.169 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:11:41.170 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.170 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:11:41.170 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:11:41.170 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:11:41.170 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.171 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.171 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:11:41.171 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:11:41.171 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:11:41.171 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.172 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.172 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:11:41.172 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:11:41.172 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:11:41.172 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.173 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.173 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.173 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.173 INFO:tasks.workunit.client.0.smithi191.stdout: "id": -4, 2021-05-12T12:11:41.173 INFO:tasks.workunit.client.0.smithi191.stdout: "name": "smithi191~hdd", 2021-05-12T12:11:41.173 INFO:tasks.workunit.client.0.smithi191.stdout: "type_id": 1, 2021-05-12T12:11:41.174 INFO:tasks.workunit.client.0.smithi191.stdout: "type_name": "host", 2021-05-12T12:11:41.174 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 19206, 2021-05-12T12:11:41.174 INFO:tasks.workunit.client.0.smithi191.stdout: "alg": "straw2", 2021-05-12T12:11:41.174 INFO:tasks.workunit.client.0.smithi191.stdout: "hash": "rjenkins1", 2021-05-12T12:11:41.174 INFO:tasks.workunit.client.0.smithi191.stdout: "items": [ 2021-05-12T12:11:41.175 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.175 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 0, 2021-05-12T12:11:41.175 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:11:41.175 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 0 2021-05-12T12:11:41.175 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.176 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.176 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 1, 2021-05-12T12:11:41.176 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:11:41.176 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 1 2021-05-12T12:11:41.176 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.177 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.177 INFO:tasks.workunit.client.0.smithi191.stdout: "id": 2, 2021-05-12T12:11:41.177 INFO:tasks.workunit.client.0.smithi191.stdout: "weight": 6402, 2021-05-12T12:11:41.177 INFO:tasks.workunit.client.0.smithi191.stdout: "pos": 2 2021-05-12T12:11:41.177 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.178 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.178 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.178 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.178 INFO:tasks.workunit.client.0.smithi191.stdout: "rules": [ 2021-05-12T12:11:41.179 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.179 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 0, 2021-05-12T12:11:41.179 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "replicated_rule", 2021-05-12T12:11:41.179 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 0, 2021-05-12T12:11:41.179 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 1, 2021-05-12T12:11:41.179 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 1, 2021-05-12T12:11:41.180 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 10, 2021-05-12T12:11:41.180 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2021-05-12T12:11:41.180 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.180 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2021-05-12T12:11:41.180 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2021-05-12T12:11:41.181 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2021-05-12T12:11:41.181 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.181 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.181 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_firstn", 2021-05-12T12:11:41.181 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2021-05-12T12:11:41.182 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2021-05-12T12:11:41.182 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.182 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.182 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2021-05-12T12:11:41.183 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.183 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.183 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.183 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.183 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_id": 1, 2021-05-12T12:11:41.184 INFO:tasks.workunit.client.0.smithi191.stdout: "rule_name": "test", 2021-05-12T12:11:41.184 INFO:tasks.workunit.client.0.smithi191.stdout: "ruleset": 1, 2021-05-12T12:11:41.184 INFO:tasks.workunit.client.0.smithi191.stdout: "type": 3, 2021-05-12T12:11:41.184 INFO:tasks.workunit.client.0.smithi191.stdout: "min_size": 3, 2021-05-12T12:11:41.184 INFO:tasks.workunit.client.0.smithi191.stdout: "max_size": 3, 2021-05-12T12:11:41.185 INFO:tasks.workunit.client.0.smithi191.stdout: "steps": [ 2021-05-12T12:11:41.185 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.185 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "set_chooseleaf_tries", 2021-05-12T12:11:41.185 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 5 2021-05-12T12:11:41.185 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.186 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.186 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "set_choose_tries", 2021-05-12T12:11:41.186 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 100 2021-05-12T12:11:41.186 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.186 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.187 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "take", 2021-05-12T12:11:41.187 INFO:tasks.workunit.client.0.smithi191.stdout: "item": -1, 2021-05-12T12:11:41.187 INFO:tasks.workunit.client.0.smithi191.stdout: "item_name": "default" 2021-05-12T12:11:41.187 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.187 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.188 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "choose_indep", 2021-05-12T12:11:41.188 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 0, 2021-05-12T12:11:41.188 INFO:tasks.workunit.client.0.smithi191.stdout: "type": "osd" 2021-05-12T12:11:41.188 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.188 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.189 INFO:tasks.workunit.client.0.smithi191.stdout: "op": "emit" 2021-05-12T12:11:41.189 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.189 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:11:41.189 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.189 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.190 INFO:tasks.workunit.client.0.smithi191.stdout: "tunables": { 2021-05-12T12:11:41.190 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_tries": 0, 2021-05-12T12:11:41.190 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:11:41.190 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_total_tries": 50, 2021-05-12T12:11:41.191 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:11:41.191 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:11:41.191 INFO:tasks.workunit.client.0.smithi191.stdout: "chooseleaf_stable": 1, 2021-05-12T12:11:41.191 INFO:tasks.workunit.client.0.smithi191.stdout: "straw_calc_version": 1, 2021-05-12T12:11:41.191 INFO:tasks.workunit.client.0.smithi191.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:11:41.192 INFO:tasks.workunit.client.0.smithi191.stdout: "profile": "jewel", 2021-05-12T12:11:41.192 INFO:tasks.workunit.client.0.smithi191.stdout: "optimal_tunables": 1, 2021-05-12T12:11:41.192 INFO:tasks.workunit.client.0.smithi191.stdout: "legacy_tunables": 0, 2021-05-12T12:11:41.192 INFO:tasks.workunit.client.0.smithi191.stdout: "minimum_required_version": "jewel", 2021-05-12T12:11:41.192 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables": 1, 2021-05-12T12:11:41.193 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables2": 1, 2021-05-12T12:11:41.193 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v2_rules": 1, 2021-05-12T12:11:41.193 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables3": 1, 2021-05-12T12:11:41.193 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v3_rules": 0, 2021-05-12T12:11:41.193 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v4_buckets": 1, 2021-05-12T12:11:41.194 INFO:tasks.workunit.client.0.smithi191.stdout: "require_feature_tunables5": 1, 2021-05-12T12:11:41.194 INFO:tasks.workunit.client.0.smithi191.stdout: "has_v5_rules": 0 2021-05-12T12:11:41.194 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.194 INFO:tasks.workunit.client.0.smithi191.stdout: "choose_args": {} 2021-05-12T12:11:41.195 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.195 INFO:tasks.workunit.client.0.smithi191.stdout: "fsmap": { 2021-05-12T12:11:41.195 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 1, 2021-05-12T12:11:41.195 INFO:tasks.workunit.client.0.smithi191.stdout: "default_fscid": -1, 2021-05-12T12:11:41.195 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": { 2021-05-12T12:11:41.196 INFO:tasks.workunit.client.0.smithi191.stdout: "compat": {}, 2021-05-12T12:11:41.196 INFO:tasks.workunit.client.0.smithi191.stdout: "ro_compat": {}, 2021-05-12T12:11:41.196 INFO:tasks.workunit.client.0.smithi191.stdout: "incompat": { 2021-05-12T12:11:41.196 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_1": "base v0.20", 2021-05-12T12:11:41.196 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:11:41.197 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:11:41.197 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:11:41.197 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:11:41.197 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:11:41.197 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_8": "no anchor table", 2021-05-12T12:11:41.198 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_9": "file layout v2", 2021-05-12T12:11:41.198 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:11:41.198 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.198 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.198 INFO:tasks.workunit.client.0.smithi191.stdout: "feature_flags": { 2021-05-12T12:11:41.199 INFO:tasks.workunit.client.0.smithi191.stdout: "enable_multiple": true, 2021-05-12T12:11:41.199 INFO:tasks.workunit.client.0.smithi191.stdout: "ever_enabled_multiple": true 2021-05-12T12:11:41.199 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.199 INFO:tasks.workunit.client.0.smithi191.stdout: "standbys": [], 2021-05-12T12:11:41.199 INFO:tasks.workunit.client.0.smithi191.stdout: "filesystems": [] 2021-05-12T12:11:41.200 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.200 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:11:41.200 INFO:tasks.workunit.client.0.smithi191.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:11:41.200 INFO:tasks.workunit.client.0.smithi191.stdout: "auth": { 2021-05-12T12:11:41.200 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:11:41.201 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 5, 2021-05-12T12:11:41.201 INFO:tasks.workunit.client.0.smithi191.stdout: "num_secrets": 3 2021-05-12T12:11:41.201 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.201 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:11:41.201 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_active": 1, 2021-05-12T12:11:41.202 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_unknown": 0, 2021-05-12T12:11:41.202 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osd": 3, 2021-05-12T12:11:41.202 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_sum": { 2021-05-12T12:11:41.202 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:11:41.202 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48430, 2021-05-12T12:11:41.203 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 101, 2021-05-12T12:11:41.203 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 1, 2021-05-12T12:11:41.203 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 303, 2021-05-12T12:11:41.203 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:11:41.203 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:11:41.204 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 101, 2021-05-12T12:11:41.204 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:11:41.204 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:11:41.204 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 101, 2021-05-12T12:11:41.204 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:11:41.205 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:11:41.205 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:11:41.205 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 101, 2021-05-12T12:11:41.205 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 101, 2021-05-12T12:11:41.205 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:11:41.206 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:11:41.206 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:11:41.206 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:11:41.206 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:11:41.206 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:11:41.207 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:11:41.207 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:11:41.207 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:11:41.207 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:11:41.207 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:11:41.208 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:11:41.208 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:11:41.208 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:11:41.208 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:11:41.209 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:11:41.209 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:11:41.209 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:11:41.209 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:11:41.209 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:11:41.210 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:11:41.210 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:11:41.210 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:11:41.210 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:11:41.210 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:11:41.211 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.211 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:11:41.211 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:11:41.211 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:11:41.211 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:11:41.212 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 0, 2021-05-12T12:11:41.212 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 0, 2021-05-12T12:11:41.212 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:11:41.212 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:41.212 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:11:41.213 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:11:41.213 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:11:41.213 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.213 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 18, 2021-05-12T12:11:41.213 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 18, 2021-05-12T12:11:41.214 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 3, 2021-05-12T12:11:41.214 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 3, 2021-05-12T12:11:41.214 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 0 2021-05-12T12:11:41.214 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.215 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum": { 2021-05-12T12:11:41.215 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:11:41.215 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:11:41.215 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:11:41.215 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:11:41.216 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:11:41.216 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:11:41.216 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:11:41.216 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 19220, 2021-05-12T12:11:41.216 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2228, 2021-05-12T12:11:41.217 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 4, 2021-05-12T12:11:41.217 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16955, 2021-05-12T12:11:41.217 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314553580, 2021-05-12T12:11:41.217 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:11:41.217 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:11:41.218 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322102865920, 2021-05-12T12:11:41.218 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:11:41.218 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2281472, 2021-05-12T12:11:41.218 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 1579657, 2021-05-12T12:11:41.218 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:11:41.219 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:41.219 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:11:41.219 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 4866, 2021-05-12T12:11:41.219 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 17362174 2021-05-12T12:11:41.219 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.220 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:11:41.220 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:11:41.220 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:11:41.220 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:11:41.220 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:11:41.221 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:11:41.221 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:11:41.221 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.221 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:11:41.221 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 0, 2021-05-12T12:11:41.222 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 0, 2021-05-12T12:11:41.222 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 0, 2021-05-12T12:11:41.222 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 0 2021-05-12T12:11:41.222 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.222 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:11:41.223 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:11:41.223 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.223 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_sum_by_class": { 2021-05-12T12:11:41.223 INFO:tasks.workunit.client.0.smithi191.stdout: "hdd": { 2021-05-12T12:11:41.223 INFO:tasks.workunit.client.0.smithi191.stdout: "up_from": 0, 2021-05-12T12:11:41.224 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 0, 2021-05-12T12:11:41.224 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pgs": 3, 2021-05-12T12:11:41.224 INFO:tasks.workunit.client.0.smithi191.stdout: "num_osds": 3, 2021-05-12T12:11:41.224 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_osds": 3, 2021-05-12T12:11:41.224 INFO:tasks.workunit.client.0.smithi191.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:11:41.225 INFO:tasks.workunit.client.0.smithi191.stdout: "kb": 314572800, 2021-05-12T12:11:41.225 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used": 19220, 2021-05-12T12:11:41.225 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_data": 2228, 2021-05-12T12:11:41.225 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_omap": 4, 2021-05-12T12:11:41.225 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_used_meta": 16955, 2021-05-12T12:11:41.226 INFO:tasks.workunit.client.0.smithi191.stdout: "kb_avail": 314553580, 2021-05-12T12:11:41.226 INFO:tasks.workunit.client.0.smithi191.stdout: "statfs": { 2021-05-12T12:11:41.226 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 322122547200, 2021-05-12T12:11:41.226 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 322102865920, 2021-05-12T12:11:41.226 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:11:41.227 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 2281472, 2021-05-12T12:11:41.227 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 1579657, 2021-05-12T12:11:41.227 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:11:41.227 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:41.227 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:11:41.228 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 4866, 2021-05-12T12:11:41.228 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 17362174 2021-05-12T12:11:41.228 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.228 INFO:tasks.workunit.client.0.smithi191.stdout: "hb_peers": [], 2021-05-12T12:11:41.228 INFO:tasks.workunit.client.0.smithi191.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:11:41.229 INFO:tasks.workunit.client.0.smithi191.stdout: "num_snap_trimming": 0, 2021-05-12T12:11:41.229 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shards_repaired": 0, 2021-05-12T12:11:41.229 INFO:tasks.workunit.client.0.smithi191.stdout: "op_queue_age_hist": { 2021-05-12T12:11:41.229 INFO:tasks.workunit.client.0.smithi191.stdout: "histogram": [], 2021-05-12T12:11:41.229 INFO:tasks.workunit.client.0.smithi191.stdout: "upper_bound": 1 2021-05-12T12:11:41.230 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.230 INFO:tasks.workunit.client.0.smithi191.stdout: "perf_stat": { 2021-05-12T12:11:41.230 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ms": 0, 2021-05-12T12:11:41.230 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ms": 0, 2021-05-12T12:11:41.230 INFO:tasks.workunit.client.0.smithi191.stdout: "commit_latency_ns": 0, 2021-05-12T12:11:41.231 INFO:tasks.workunit.client.0.smithi191.stdout: "apply_latency_ns": 0 2021-05-12T12:11:41.231 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.231 INFO:tasks.workunit.client.0.smithi191.stdout: "alerts": [], 2021-05-12T12:11:41.231 INFO:tasks.workunit.client.0.smithi191.stdout: "network_ping_times": [] 2021-05-12T12:11:41.231 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.232 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.232 INFO:tasks.workunit.client.0.smithi191.stdout: "pool_stats": [ 2021-05-12T12:11:41.232 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.232 INFO:tasks.workunit.client.0.smithi191.stdout: "poolid": 1, 2021-05-12T12:11:41.232 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg": 1, 2021-05-12T12:11:41.233 INFO:tasks.workunit.client.0.smithi191.stdout: "stat_sum": { 2021-05-12T12:11:41.233 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes": 48430, 2021-05-12T12:11:41.233 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects": 101, 2021-05-12T12:11:41.233 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_clones": 1, 2021-05-12T12:11:41.233 INFO:tasks.workunit.client.0.smithi191.stdout: "num_object_copies": 303, 2021-05-12T12:11:41.234 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:11:41.234 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_missing": 0, 2021-05-12T12:11:41.234 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_degraded": 101, 2021-05-12T12:11:41.234 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_misplaced": 0, 2021-05-12T12:11:41.235 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_unfound": 0, 2021-05-12T12:11:41.235 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_dirty": 101, 2021-05-12T12:11:41.235 INFO:tasks.workunit.client.0.smithi191.stdout: "num_whiteouts": 0, 2021-05-12T12:11:41.235 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read": 0, 2021-05-12T12:11:41.235 INFO:tasks.workunit.client.0.smithi191.stdout: "num_read_kb": 0, 2021-05-12T12:11:41.236 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write": 101, 2021-05-12T12:11:41.236 INFO:tasks.workunit.client.0.smithi191.stdout: "num_write_kb": 101, 2021-05-12T12:11:41.236 INFO:tasks.workunit.client.0.smithi191.stdout: "num_scrub_errors": 0, 2021-05-12T12:11:41.236 INFO:tasks.workunit.client.0.smithi191.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:11:41.236 INFO:tasks.workunit.client.0.smithi191.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:11:41.237 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_recovered": 0, 2021-05-12T12:11:41.237 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_recovered": 0, 2021-05-12T12:11:41.237 INFO:tasks.workunit.client.0.smithi191.stdout: "num_keys_recovered": 0, 2021-05-12T12:11:41.237 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_omap": 0, 2021-05-12T12:11:41.238 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:11:41.238 INFO:tasks.workunit.client.0.smithi191.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:11:41.238 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush": 0, 2021-05-12T12:11:41.238 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_kb": 0, 2021-05-12T12:11:41.239 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict": 0, 2021-05-12T12:11:41.239 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_kb": 0, 2021-05-12T12:11:41.239 INFO:tasks.workunit.client.0.smithi191.stdout: "num_promote": 0, 2021-05-12T12:11:41.239 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_high": 0, 2021-05-12T12:11:41.239 INFO:tasks.workunit.client.0.smithi191.stdout: "num_flush_mode_low": 0, 2021-05-12T12:11:41.240 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_some": 0, 2021-05-12T12:11:41.240 INFO:tasks.workunit.client.0.smithi191.stdout: "num_evict_mode_full": 0, 2021-05-12T12:11:41.240 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_pinned": 0, 2021-05-12T12:11:41.240 INFO:tasks.workunit.client.0.smithi191.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:11:41.241 INFO:tasks.workunit.client.0.smithi191.stdout: "num_large_omap_objects": 0, 2021-05-12T12:11:41.241 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_manifest": 0, 2021-05-12T12:11:41.241 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_bytes": 0, 2021-05-12T12:11:41.241 INFO:tasks.workunit.client.0.smithi191.stdout: "num_omap_keys": 0, 2021-05-12T12:11:41.242 INFO:tasks.workunit.client.0.smithi191.stdout: "num_objects_repaired": 0 2021-05-12T12:11:41.242 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.242 INFO:tasks.workunit.client.0.smithi191.stdout: "store_stats": { 2021-05-12T12:11:41.242 INFO:tasks.workunit.client.0.smithi191.stdout: "total": 0, 2021-05-12T12:11:41.243 INFO:tasks.workunit.client.0.smithi191.stdout: "available": 0, 2021-05-12T12:11:41.243 INFO:tasks.workunit.client.0.smithi191.stdout: "internally_reserved": 0, 2021-05-12T12:11:41.243 INFO:tasks.workunit.client.0.smithi191.stdout: "allocated": 1241088, 2021-05-12T12:11:41.243 INFO:tasks.workunit.client.0.smithi191.stdout: "data_stored": 1241088, 2021-05-12T12:11:41.244 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed": 0, 2021-05-12T12:11:41.244 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:41.244 INFO:tasks.workunit.client.0.smithi191.stdout: "data_compressed_original": 0, 2021-05-12T12:11:41.244 INFO:tasks.workunit.client.0.smithi191.stdout: "omap_allocated": 0, 2021-05-12T12:11:41.244 INFO:tasks.workunit.client.0.smithi191.stdout: "internal_metadata": 0 2021-05-12T12:11:41.245 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.245 INFO:tasks.workunit.client.0.smithi191.stdout: "log_size": 18, 2021-05-12T12:11:41.245 INFO:tasks.workunit.client.0.smithi191.stdout: "ondisk_log_size": 18, 2021-05-12T12:11:41.245 INFO:tasks.workunit.client.0.smithi191.stdout: "up": 3, 2021-05-12T12:11:41.245 INFO:tasks.workunit.client.0.smithi191.stdout: "acting": 3, 2021-05-12T12:11:41.246 INFO:tasks.workunit.client.0.smithi191.stdout: "num_store_stats": 3 2021-05-12T12:11:41.246 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.246 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.246 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_stats": [ 2021-05-12T12:11:41.246 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.247 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:11:41.247 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 184683593738 2021-05-12T12:11:41.247 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.247 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.248 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:11:41.248 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 42949672982 2021-05-12T12:11:41.248 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.248 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.248 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:11:41.249 INFO:tasks.workunit.client.0.smithi191.stdout: "seq": 201863462922 2021-05-12T12:11:41.249 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.249 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.249 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_state": [ 2021-05-12T12:11:41.249 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.250 INFO:tasks.workunit.client.0.smithi191.stdout: "state": "active+undersized+degraded", 2021-05-12T12:11:41.250 INFO:tasks.workunit.client.0.smithi191.stdout: "num": 1 2021-05-12T12:11:41.250 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.250 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.250 INFO:tasks.workunit.client.0.smithi191.stdout: "num_pg_by_osd": [ 2021-05-12T12:11:41.251 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.251 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2, 2021-05-12T12:11:41.251 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:11:41.251 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:11:41.251 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:41.252 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.252 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.252 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 2147483647, 2021-05-12T12:11:41.252 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:11:41.252 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:11:41.253 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:41.253 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.253 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.253 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 1, 2021-05-12T12:11:41.253 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 0, 2021-05-12T12:11:41.254 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 0, 2021-05-12T12:11:41.254 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:41.254 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.254 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.254 INFO:tasks.workunit.client.0.smithi191.stdout: "osd": 0, 2021-05-12T12:11:41.255 INFO:tasks.workunit.client.0.smithi191.stdout: "num_primary_pg": 1, 2021-05-12T12:11:41.255 INFO:tasks.workunit.client.0.smithi191.stdout: "num_acting_pg": 1, 2021-05-12T12:11:41.255 INFO:tasks.workunit.client.0.smithi191.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:41.255 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.256 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.256 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": [ 2021-05-12T12:11:41.256 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:11:41.256 INFO:tasks.workunit.client.0.smithi191.stdout: "pool": 1, 2021-05-12T12:11:41.256 INFO:tasks.workunit.client.0.smithi191.stdout: "purged_snaps": {} 2021-05-12T12:11:41.257 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.257 INFO:tasks.workunit.client.0.smithi191.stdout: ], 2021-05-12T12:11:41.257 INFO:tasks.workunit.client.0.smithi191.stdout: "servicemap": { 2021-05-12T12:11:41.257 INFO:tasks.workunit.client.0.smithi191.stdout: "epoch": 5, 2021-05-12T12:11:41.257 INFO:tasks.workunit.client.0.smithi191.stdout: "modified": "2021-05-12T12:10:06.651673+0000", 2021-05-12T12:11:41.258 INFO:tasks.workunit.client.0.smithi191.stdout: "services": {} 2021-05-12T12:11:41.258 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:11:41.258 INFO:tasks.workunit.client.0.smithi191.stdout: "paxos": { 2021-05-12T12:11:41.258 INFO:tasks.workunit.client.0.smithi191.stdout: "first_committed": 1, 2021-05-12T12:11:41.259 INFO:tasks.workunit.client.0.smithi191.stdout: "last_committed": 533, 2021-05-12T12:11:41.259 INFO:tasks.workunit.client.0.smithi191.stdout: "last_pn": 0, 2021-05-12T12:11:41.259 INFO:tasks.workunit.client.0.smithi191.stdout: "accepted_pn": 0 2021-05-12T12:11:41.259 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:11:41.259 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:11:41.260 INFO:tasks.workunit.client.0.smithi191.stderr:report 1136468323 2021-05-12T12:11:41.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:11:41.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2021-05-12T12:11:41.261 INFO:tasks.workunit.client.0.smithi191.stdout:delay recovery 2021-05-12T12:11:41.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:11:41.354 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:11:41.355 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:06.467837+0000 40'102 48:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:11:41.355 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:11:41.355 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:11:41.356 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:11:41.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:11:41.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2021-05-12T12:11:41.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:11:41.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:11:41.368 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:11:41.368 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:41.368 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:11:41.368 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.0.asok 2021-05-12T12:11:41.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:11:41.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T12:11:41.462 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:11:41.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:11:41.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2021-05-12T12:11:41.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:11:41.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:11:41.472 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:11:41.472 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:41.472 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:11:41.473 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.2.asok 2021-05-12T12:11:41.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:11:41.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T12:11:41.567 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:11:41.575 INFO:tasks.workunit.client.0.smithi191.stdout:revive divergent 1 2021-05-12T12:11:41.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2021-05-12T12:11:41.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:11:41.916 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:11:41.916 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:06.467837+0000 40'102 48:135 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:11:41.917 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:11:41.917 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:11:41.918 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:11:41.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2021-05-12T12:11:42.770 INFO:tasks.workunit.client.0.smithi191.stderr:noup is set 2021-05-12T12:11:42.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-05-12T12:11:42.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:11:42.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:11:42.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:11:42.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:11:42.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:11:42.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:11:42.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:11:42.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:11:42.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:11:42.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:11:42.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:11:42.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:11:42.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:11:42.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:11:42.785 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:11:42.786 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:11:42.786 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:11:42.786 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:11:42.786 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:42.787 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:11:42.787 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:11:42.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:11:42.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:11:42.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:11:42.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:11:42.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:11:42.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:11:42.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:42.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:11:42.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:11:42.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:11:42.789 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:11:42.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:11:42.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:42.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:11:42.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:11:42.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:11:42.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:11:42.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:11:42.811 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:42.808+0000 7f1c9f2c70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:42.812 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:42.810+0000 7f1c9f2c70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:42.814 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:42.811+0000 7f1c9f2c70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:43.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T12:11:43.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2021-05-12T12:11:43.333 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:43.330+0000 7f1c9f2c70c0 -1 Falling back to public interface 2021-05-12T12:11:45.665 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:11:45.662+0000 7f1c9f2c70c0 -1 osd.1 28 log_to_monitors {default=true} 2021-05-12T12:11:48.153 INFO:tasks.workunit.client.0.smithi191.stdout:delay recovery divergent 2021-05-12T12:11:48.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2021-05-12T12:11:48.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:11:48.505 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:11:48.505 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:06.467837+0000 40'102 48:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:11:48.506 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:11:48.506 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:11:48.507 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:11:48.517 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2021-05-12T12:11:48.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:11:48.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:11:48.518 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:11:48.518 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:48.518 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:11:48.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34183/ceph-osd.1.asok 2021-05-12T12:11:48.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:11:48.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.34183/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T12:11:48.612 INFO:tasks.workunit.client.0.smithi191.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:11:48.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2021-05-12T12:11:49.289 INFO:tasks.workunit.client.0.smithi191.stderr:noup is unset 2021-05-12T12:11:49.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2021-05-12T12:11:49.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:11:49.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:11:49.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:11:49.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:11:49.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:11:49.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:11:49.300 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:11:49.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:11:49.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:11:49.682 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 43 up_thru 47 down_at 42 last_clean_interval [35,40) [v2:127.0.0.1:6802/84260,v1:127.0.0.1:6803/84260] [v2:127.0.0.1:6804/84260,v1:127.0.0.1:6805/84260] exists,up 3164e607-150e-4cc6-bf11-92a6263c5c19 2021-05-12T12:11:49.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:11:49.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:11:49.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:11:49.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2021-05-12T12:11:49.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:11:49.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:11:49.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:11:49.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:11:49.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:11:49.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:11:49.686 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:11:49.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:11:49.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:11:50.060 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/87111,v1:127.0.0.1:6819/87111] [v2:127.0.0.1:6820/87111,v1:127.0.0.1:6821/87111] exists,up 4249e897-1cf7-49ad-8d28-f401f2373f5a 2021-05-12T12:11:50.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:11:50.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:11:50.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:11:50.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2021-05-12T12:11:50.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:11:50.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:11:50.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:11:50.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:11:50.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:11:50.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:11:50.064 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:11:50.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:11:50.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:11:50.427 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 47 up_thru 0 down_at 46 last_clean_interval [37,40) [v2:127.0.0.1:6810/84943,v1:127.0.0.1:6811/84943] [v2:127.0.0.1:6812/84943,v1:127.0.0.1:6813/84943] exists,up d6a6ae15-1b91-42f9-8ba3-5cd4469c3918 2021-05-12T12:11:50.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:11:50.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:11:50.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:11:50.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:11:50.802 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:11:50.802 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:06.467837+0000 40'102 50:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:11:50.802 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:11:50.803 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:11:50.804 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:11:50.813 INFO:tasks.workunit.client.0.smithi191.stdout:wait for peering 2021-05-12T12:11:50.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2021-05-12T12:11:50.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:11:51.157 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:11:51.157 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:06.467837+0000 40'102 50:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:11:51.157 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:11:51.158 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:11:51.159 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:11:51.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2021-05-12T12:12:02.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2021-05-12T12:12:02.586 INFO:tasks.workunit.client.0.smithi191.stdout:killing divergent 1 2021-05-12T12:12:02.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:02.934 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:02.935 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2021-05-12T12:11:49.931051+0000 40'102 54:128 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:12:02.935 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:12:02.935 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:02.936 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:12:02.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2021-05-12T12:12:02.946 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:02.946 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:02.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:02.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:02.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:03.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:03.051 INFO:tasks.workunit.client.0.smithi191.stdout:reviving divergent 1 2021-05-12T12:12:03.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2021-05-12T12:12:03.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:03.403 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:03.404 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2021-05-12T12:11:49.931051+0000 40'102 54:128 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:12:03.404 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:12:03.404 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:03.405 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:12:03.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-05-12T12:12:03.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:12:03.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:12:03.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:12:03.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:12:03.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:12:03.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:12:03.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:12:03.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:12:03.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:12:03.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:12:03.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:12:03.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:12:03.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:12:03.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:12:03.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:12:03.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:03.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:03.422 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:03.422 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:03.422 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:12:03.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:12:03.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' 2021-05-12T12:12:03.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:12:03.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:12:03.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:12:03.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:12:03.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:12:03.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:03.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:12:03.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:12:03.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:12:03.425 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:12:03.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:12:03.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=58a97f67-bf94-4196-a5b7-5319dbe39c36 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.34183/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:03.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:12:03.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:12:03.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:12:03.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:12:03.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:12:03.445 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:03.442+0000 7f2ab2ca50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:03.447 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:03.444+0000 7f2ab2ca50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:03.448 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:03.445+0000 7f2ab2ca50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:03.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:12:03.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:03.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:12:03.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:03.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:03.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:03.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:03.789 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:12:03.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:03.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:12:03.965 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:03.962+0000 7f2ab2ca50c0 -1 Falling back to public interface 2021-05-12T12:12:04.180 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/87111,v1:127.0.0.1:6819/87111] [v2:127.0.0.1:6820/87111,v1:127.0.0.1:6821/87111] exists,up 4249e897-1cf7-49ad-8d28-f401f2373f5a 2021-05-12T12:12:04.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:12:04.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:12:04.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:12:04.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2021-05-12T12:12:06.520 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:06.517+0000 7f2ab2ca50c0 -1 osd.1 54 log_to_monitors {default=true} 2021-05-12T12:12:24.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2021-05-12T12:12:24.182 INFO:tasks.workunit.client.0.smithi191.stdout:allowing recovery 2021-05-12T12:12:24.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:24.539 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:24.539 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2021-05-12T12:12:14.939177+0000 54'103 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:12:24.540 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:12:24.540 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:24.541 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:12:24.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-05-12T12:12:24.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T12:12:24.674 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:12:24.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-05-12T12:12:24.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T12:12:24.819 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:12:24.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-05-12T12:12:24.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T12:12:24.951 INFO:tasks.workunit.client.0.smithi191.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:12:24.960 INFO:tasks.workunit.client.0.smithi191.stdout:reading divergent objects 2021-05-12T12:12:24.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2021-05-12T12:12:24.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:25.307 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:25.308 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2021-05-12T12:12:14.939177+0000 54'103 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:07.750558+0000 0'0 2021-05-12T12:09:07.750558+0000 0 2021-05-12T12:12:25.308 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:12:25.308 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:25.309 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:12:25.319 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2021-05-12T12:12:25.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2021-05-12T12:12:25.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:12:25.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T12:12:25.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:12:25.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T12:12:25.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:12:25.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T12:12:25.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:12:25.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T12:12:25.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2021-05-12T12:12:25.461 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-05-12T12:12:25.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:12:25.466 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 28'103 (27'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-05-12T12:09:46.015637+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:12:25.466 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 20 _merge_object_divergent_entries: keeping 28'103 (27'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-05-12T12:09:46.015637+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:12:25.466 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:12:25.466 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 27'99 first_divergent_update: 28'103 last_divergent_update: 28'103 2021-05-12T12:12:25.467 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2021-05-12T12:12:25.467 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2021-05-12T12:12:25.467 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4565.0:1 2021-05-12T12:09:53.347081+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:25.467 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 20 _merge_object_divergent_entries: keeping 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4565.0:1 2021-05-12T12:09:53.347081+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:25.467 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:12:25.468 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 27'99 first_divergent_update: 28'104 last_divergent_update: 28'104 2021-05-12T12:12:25.468 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2021-05-12T12:12:25.468 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:12:25.468 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 28'104 2021-05-12T12:12:25.468 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4565.0:1 2021-05-12T12:09:53.347081+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:25.469 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2021-05-12T12:12:25.469 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4556.0:1 2021-05-12T12:09:52.129157+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4562.0:1 2021-05-12T12:09:53.148773+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:25.469 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 20 _merge_object_divergent_entries: keeping 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4556.0:1 2021-05-12T12:09:52.129157+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:25.470 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 20 _merge_object_divergent_entries: keeping 27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4562.0:1 2021-05-12T12:09:53.148773+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:25.470 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:12:25.470 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 27'100 first_divergent_update: 27'101 last_divergent_update: 27'102 2021-05-12T12:12:25.470 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2021-05-12T12:12:25.470 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:12:25.471 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'104 2021-05-12T12:12:25.471 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4562.0:1 2021-05-12T12:09:53.148773+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:25.471 INFO:tasks.workunit.client.0.smithi191.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:11:49.643+0000 7f1c7b1bd700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-05-12T12:12:25.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-05-12T12:12:25.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-05-12T12:12:25.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2021-05-12T12:12:25.474 INFO:tasks.workunit.client.0.smithi191.stdout:success 2021-05-12T12:12:25.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2021-05-12T12:12:25.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T12:12:25.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T12:12:25.964 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test' does not exist 2021-05-12T12:12:25.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2021-05-12T12:12:25.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:25.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:25.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:25.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:25.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:26.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:26.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T12:12:26.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:12:26.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:12:26.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:12:26.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:26.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:26.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:26.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:26.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:26.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:26.088 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:12:26.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:12:26.089 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:12:26.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:12:26.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:12:26.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:12:26.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:26.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:12:26.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:12:26.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:26.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:12:26.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:12:26.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:12:26.107 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:12:26.108 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:26.108 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:12:26.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34183 2021-05-12T12:12:26.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:12:26.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:12:26.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:12:26.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/divergent-priors 0 2021-05-12T12:12:26.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:12:26.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:12:26.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:12:26.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:26.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:26.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:26.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:26.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:26.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:26.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:12:26.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:12:26.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:12:26.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:12:26.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:12:26.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:12:26.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:26.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:12:26.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:12:26.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:26.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:12:26.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:12:26.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:12:26.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:12:26.119 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:26.119 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34183 2021-05-12T12:12:26.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34183 2021-05-12T12:12:26.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:12:26.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:12:26.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:12:26.122 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:12:26.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:12:26.156 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2021-05-12T12:12:26.157 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2021-05-12T12:12:26.217 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:12:26.225 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:12:26.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:12:26.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:12:26.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:12:26.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:12:26.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:12:26.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:12:26.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:12:26.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:12:26.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/ec-error-rollforward 2021-05-12T12:12:26.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2021-05-12T12:12:26.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2021-05-12T12:12:26.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2021-05-12T12:12:26.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2021-05-12T12:12:26.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2021-05-12T12:12:26.228 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2021-05-12T12:12:26.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none ' 2021-05-12T12:12:26.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2021-05-12T12:12:26.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2021-05-12T12:12:26.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2021-05-12T12:12:26.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2021-05-12T12:12:26.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2021-05-12T12:12:26.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2021-05-12T12:12:26.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2021-05-12T12:12:26.230 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2021-05-12T12:12:26.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:12:26.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2021-05-12T12:12:26.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2021-05-12T12:12:26.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2021-05-12T12:12:26.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2021-05-12T12:12:26.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2021-05-12T12:12:26.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-05-12T12:12:26.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:12:26.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-05-12T12:12:26.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:26.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:26.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:26.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:26.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:26.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:26.237 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:12:26.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:12:26.237 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:12:26.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:12:26.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:12:26.238 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:12:26.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:26.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:12:26.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:12:26.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:26.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:12:26.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:12:26.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-05-12T12:12:26.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:12:26.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:26.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:26.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.88679 2021-05-12T12:12:26.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:12:26.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:12:26.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2021-05-12T12:12:26.244 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:12:26.244 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:26.244 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:26.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.88679 2021-05-12T12:12:26.244 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:12:26.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:12:26.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:12:26.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:12:26.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2021-05-12T12:12:26.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2021-05-12T12:12:26.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2021-05-12T12:12:26.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2021-05-12T12:12:26.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2021-05-12T12:12:26.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:12:26.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:12:26.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:12:26.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2021-05-12T12:12:26.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2021-05-12T12:12:26.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:12:26.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:26.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:26.603 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:26.604 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:26.604 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:26.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:26.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:12:26.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:12:26.882 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:12:26.882 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:12:26.882 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:12:26.882 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:12:26.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:12:26.883 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:12:26.883 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:12:26.883 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:12:26.883 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:12:26.884 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:26.884 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:26.884 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.88679/ceph-mon.a.asok 2021-05-12T12:12:26.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:12:26.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.88679/ceph-mon.a.asok config get fsid 2021-05-12T12:12:26.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:12:26.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:12:26.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:12:26.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:12:26.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:12:26.978 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:12:26.978 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:12:26.979 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:12:26.979 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:12:26.979 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:26.979 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:26.979 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.88679/ceph-mon.a.asok 2021-05-12T12:12:26.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:12:26.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.88679/ceph-mon.a.asok config get mon_host 2021-05-12T12:12:27.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2021-05-12T12:12:27.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2021-05-12T12:12:27.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:12:27.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:12:27.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:12:27.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2021-05-12T12:12:27.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:12:27.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:12:27.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:27.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:27.920 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:27.921 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:27.921 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:27.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:27.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:12:27.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:12:27.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2021-05-12T12:12:27.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T12:12:27.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:12:27.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:12:27.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:12:27.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2021-05-12T12:12:27.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:12:27.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:12:27.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:12:27.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:12:27.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-05-12T12:12:27.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-05-12T12:12:27.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:12:27.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:12:27.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:12:27.946 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:12:27.946 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:27.946 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:27.946 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:27.947 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:27.947 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:27.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:27.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:27.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:12:27.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:12:27.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:12:27.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:12:27.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:12:27.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:12:27.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:12:27.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:27.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:12:27.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:12:27.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/0 2021-05-12T12:12:27.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:12:27.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=9e938bc4-7017-45f8-99dd-cea127023b11 2021-05-12T12:12:27.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 9e938bc4-7017-45f8-99dd-cea127023b11' 2021-05-12T12:12:27.950 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 9e938bc4-7017-45f8-99dd-cea127023b11 2021-05-12T12:12:27.951 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:12:27.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQArxptg3m3UOBAAlYVWhNC5VpzzuudsSA76BA== 2021-05-12T12:12:27.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQArxptg3m3UOBAAlYVWhNC5VpzzuudsSA76BA=="}' 2021-05-12T12:12:27.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 9e938bc4-7017-45f8-99dd-cea127023b11 -i td/ec-error-rollforward/0/new.json 2021-05-12T12:12:28.309 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:12:28.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/0/new.json 2021-05-12T12:12:28.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQArxptg3m3UOBAAlYVWhNC5VpzzuudsSA76BA== --osd-uuid 9e938bc4-7017-45f8-99dd-cea127023b11 2021-05-12T12:12:28.347 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:28.344+0000 7f989d4990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:28.353 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:28.351+0000 7f989d4990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:28.358 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:28.353+0000 7f989d4990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:28.358 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:28.354+0000 7f989d4990c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-05-12T12:12:28.359 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:28.354+0000 7f989d4990c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-05-12T12:12:28.359 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:28.354+0000 7f989d4990c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-05-12T12:12:28.397 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:28.394+0000 7f989d4990c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2021-05-12T12:12:33.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2021-05-12T12:12:33.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:12:33.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:12:33.533 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:12:33.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:12:34.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:12:34.021 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:12:34.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:34.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:12:34.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:12:34.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:12:34.039 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:34.037+0000 7f55af8990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:34.041 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:34.038+0000 7f55af8990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:34.042 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:34.039+0000 7f55af8990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:34.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:12:34.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:34.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:12:34.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:34.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:34.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:34.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:34.381 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:12:34.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:34.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:12:34.559 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:34.556+0000 7f55af8990c0 -1 Falling back to public interface 2021-05-12T12:12:34.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:35.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:35.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:35.749 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:12:35.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:12:35.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:35.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:12:36.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:36.747 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:36.744+0000 7f55af8990c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:12:37.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:37.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:37.116 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:12:37.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:12:37.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:37.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:12:37.487 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/89799,v1:127.0.0.1:6803/89799] [v2:127.0.0.1:6804/89799,v1:127.0.0.1:6805/89799] exists,up 9e938bc4-7017-45f8-99dd-cea127023b11 2021-05-12T12:12:37.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:12:37.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:12:37.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:12:37.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2021-05-12T12:12:37.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T12:12:37.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:12:37.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:12:37.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:12:37.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2021-05-12T12:12:37.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:12:37.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:12:37.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:12:37.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:12:37.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-05-12T12:12:37.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-05-12T12:12:37.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:12:37.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:12:37.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:12:37.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:12:37.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:37.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:37.496 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:37.496 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:37.496 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:37.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:37.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:37.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:12:37.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:12:37.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:12:37.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:12:37.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:12:37.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:12:37.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:12:37.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:37.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:12:37.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:12:37.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/1 2021-05-12T12:12:37.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:12:37.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=42874c9e-3714-454d-93d6-1a7a3e1423dc 2021-05-12T12:12:37.500 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 42874c9e-3714-454d-93d6-1a7a3e1423dc 2021-05-12T12:12:37.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 42874c9e-3714-454d-93d6-1a7a3e1423dc' 2021-05-12T12:12:37.501 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:12:37.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA1xptgeaPQHRAApsNjdDIrTcafPkwLrDGkdg== 2021-05-12T12:12:37.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA1xptgeaPQHRAApsNjdDIrTcafPkwLrDGkdg=="}' 2021-05-12T12:12:37.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 42874c9e-3714-454d-93d6-1a7a3e1423dc -i td/ec-error-rollforward/1/new.json 2021-05-12T12:12:37.958 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:12:37.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/1/new.json 2021-05-12T12:12:37.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA1xptgeaPQHRAApsNjdDIrTcafPkwLrDGkdg== --osd-uuid 42874c9e-3714-454d-93d6-1a7a3e1423dc 2021-05-12T12:12:37.991 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:37.988+0000 7f76676ea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:37.992 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:37.989+0000 7f76676ea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:37.993 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:37.990+0000 7f76676ea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:37.994 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:37.991+0000 7f76676ea0c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-05-12T12:12:37.994 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:37.991+0000 7f76676ea0c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-05-12T12:12:38.000 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:37.991+0000 7f76676ea0c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-05-12T12:12:38.044 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:38.041+0000 7f76676ea0c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2021-05-12T12:12:42.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2021-05-12T12:12:42.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:12:42.995 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:12:42.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:12:42.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:12:43.479 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:12:43.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:12:43.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:43.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:12:43.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:12:43.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:12:43.498 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:43.495+0000 7f2f420ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:43.500 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:43.497+0000 7f2f420ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:43.502 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:43.498+0000 7f2f420ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:43.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:12:43.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:43.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:12:43.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:43.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:43.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:43.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:43.837 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:12:43.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:43.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:12:44.017 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:44.015+0000 7f2f420ef0c0 -1 Falling back to public interface 2021-05-12T12:12:44.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:45.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:45.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:45.202 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:12:45.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:12:45.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:45.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:12:45.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:46.241 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:46.238+0000 7f2f420ef0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:12:46.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:46.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:46.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:12:46.569 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:12:46.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:46.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:12:46.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:47.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:47.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:47.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:12:47.928 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:12:47.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:47.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:12:48.282 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/91184,v1:127.0.0.1:6811/91184] [v2:127.0.0.1:6812/91184,v1:127.0.0.1:6813/91184] exists,up 42874c9e-3714-454d-93d6-1a7a3e1423dc 2021-05-12T12:12:48.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:12:48.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:12:48.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:12:48.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2021-05-12T12:12:48.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T12:12:48.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:12:48.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:12:48.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:12:48.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2021-05-12T12:12:48.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:12:48.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:12:48.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:12:48.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:12:48.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-05-12T12:12:48.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-05-12T12:12:48.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:12:48.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:12:48.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:12:48.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:12:48.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:48.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:48.291 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:48.291 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:48.292 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:48.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:48.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:48.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:12:48.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:12:48.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:12:48.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:12:48.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:12:48.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:12:48.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:12:48.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:48.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:12:48.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:12:48.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/2 2021-05-12T12:12:48.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:12:48.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=bf730840-416a-44da-9293-d75232e29834 2021-05-12T12:12:48.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 bf730840-416a-44da-9293-d75232e29834' 2021-05-12T12:12:48.295 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 bf730840-416a-44da-9293-d75232e29834 2021-05-12T12:12:48.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:12:48.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBAxptgSOG7ERAAl8iyaxLUnt8WDlJiqJqj9w== 2021-05-12T12:12:48.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBAxptgSOG7ERAAl8iyaxLUnt8WDlJiqJqj9w=="}' 2021-05-12T12:12:48.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new bf730840-416a-44da-9293-d75232e29834 -i td/ec-error-rollforward/2/new.json 2021-05-12T12:12:48.804 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:12:48.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/2/new.json 2021-05-12T12:12:48.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBAxptgSOG7ERAAl8iyaxLUnt8WDlJiqJqj9w== --osd-uuid bf730840-416a-44da-9293-d75232e29834 2021-05-12T12:12:48.837 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:48.834+0000 7f9208d970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:48.839 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:48.836+0000 7f9208d970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:48.841 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:48.838+0000 7f9208d970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:48.841 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:48.838+0000 7f9208d970c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-05-12T12:12:48.842 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:48.838+0000 7f9208d970c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-05-12T12:12:48.842 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:48.838+0000 7f9208d970c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-05-12T12:12:48.874 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:48.871+0000 7f9208d970c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2021-05-12T12:12:53.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2021-05-12T12:12:53.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:12:53.762 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:12:53.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:12:53.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:12:54.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:12:54.245 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:12:54.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:54.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:12:54.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:12:54.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:12:54.264 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:54.261+0000 7ff4a97450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:54.265 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:54.263+0000 7ff4a97450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:54.267 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:54.264+0000 7ff4a97450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:54.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:12:54.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:54.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:12:54.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:54.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:54.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:54.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:54.614 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:12:54.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:54.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:12:54.783 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:54.780+0000 7ff4a97450c0 -1 Falling back to public interface 2021-05-12T12:12:54.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:55.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:55.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:55.974 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:12:55.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:12:55.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:55.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:12:56.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:57.117 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:57.114+0000 7ff4a97450c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:12:57.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:57.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:57.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:12:57.340 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:12:57.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:57.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:12:57.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:58.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:58.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:58.816 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:12:58.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:12:58.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:58.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:12:59.173 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/92595,v1:127.0.0.1:6819/92595] [v2:127.0.0.1:6820/92595,v1:127.0.0.1:6821/92595] exists,up bf730840-416a-44da-9293-d75232e29834 2021-05-12T12:12:59.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:12:59.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:12:59.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:12:59.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2021-05-12T12:12:59.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T12:12:59.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:12:59.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-05-12T12:12:59.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:12:59.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2021-05-12T12:12:59.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:12:59.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:12:59.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:12:59.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:12:59.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-05-12T12:12:59.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-05-12T12:12:59.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:12:59.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:12:59.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:12:59.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:12:59.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:59.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:59.182 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:59.182 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:59.182 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:12:59.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:59.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:12:59.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:12:59.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:12:59.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:12:59.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:12:59.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:12:59.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:12:59.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:12:59.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:59.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:12:59.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:12:59.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/3 2021-05-12T12:12:59.186 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:12:59.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=8ec65632-58f0-4715-a14d-e33f833d1811 2021-05-12T12:12:59.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 8ec65632-58f0-4715-a14d-e33f833d1811' 2021-05-12T12:12:59.186 INFO:tasks.workunit.client.0.smithi191.stdout:add osd3 8ec65632-58f0-4715-a14d-e33f833d1811 2021-05-12T12:12:59.187 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:12:59.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBLxptgDhwvCxAAX6uyA1AIiWVNHmT3J70Piw== 2021-05-12T12:12:59.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBLxptgDhwvCxAAX6uyA1AIiWVNHmT3J70Piw=="}' 2021-05-12T12:12:59.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 8ec65632-58f0-4715-a14d-e33f833d1811 -i td/ec-error-rollforward/3/new.json 2021-05-12T12:12:59.639 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:12:59.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/3/new.json 2021-05-12T12:12:59.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBLxptgDhwvCxAAX6uyA1AIiWVNHmT3J70Piw== --osd-uuid 8ec65632-58f0-4715-a14d-e33f833d1811 2021-05-12T12:12:59.672 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:59.669+0000 7fd045ff90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:59.674 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:59.671+0000 7fd045ff90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:59.675 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:59.672+0000 7fd045ff90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:59.676 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:59.673+0000 7fd045ff90c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-05-12T12:12:59.676 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:59.673+0000 7fd045ff90c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-05-12T12:12:59.676 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:59.673+0000 7fd045ff90c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-05-12T12:12:59.716 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:12:59.713+0000 7fd045ff90c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2021-05-12T12:13:04.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2021-05-12T12:13:04.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:13:04.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-05-12T12:13:04.701 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd3 key to auth repository 2021-05-12T12:13:04.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:13:05.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-05-12T12:13:05.214 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.3 2021-05-12T12:13:05.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:05.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:13:05.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:13:05.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:13:05.232 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:05.229+0000 7f699ad170c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:05.234 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:05.231+0000 7f699ad170c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:05.235 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:05.232+0000 7f699ad170c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:05.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-05-12T12:13:05.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:05.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:13:05.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:05.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:05.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:05.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:05.575 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:13:05.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:05.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:13:05.751 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:05.748+0000 7f699ad170c0 -1 Falling back to public interface 2021-05-12T12:13:05.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:06.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:06.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:06.935 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:13:06.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:13:06.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:06.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:13:07.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:07.965 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:07.962+0000 7f699ad170c0 -1 osd.3 0 log_to_monitors {default=true} 2021-05-12T12:13:08.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:08.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:08.306 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:13:08.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:13:08.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:08.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:13:08.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:09.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:09.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:09.678 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:13:09.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:13:09.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:09.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:13:10.064 INFO:tasks.workunit.client.0.smithi191.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/94006,v1:127.0.0.1:6827/94006] [v2:127.0.0.1:6828/94006,v1:127.0.0.1:6829/94006] exists,up 8ec65632-58f0-4715-a14d-e33f833d1811 2021-05-12T12:13:10.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:10.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:10.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:10.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2021-05-12T12:13:10.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2021-05-12T12:13:11.236 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'ec' already exists 2021-05-12T12:13:11.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2021-05-12T12:13:13.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2021-05-12T12:13:13.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 92595 2021-05-12T12:13:13.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=94644 2021-05-12T12:13:13.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2021-05-12T12:13:13.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2021-05-12T12:13:14.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 94665' 2021-05-12T12:13:14.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2021-05-12T12:13:14.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 94666' 2021-05-12T12:13:14.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2021-05-12T12:13:14.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 94667' 2021-05-12T12:13:14.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2021-05-12T12:13:14.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2021-05-12T12:13:15.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2021-05-12T12:13:15.370 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:13:15.370 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:13:15.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:13:15.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:13:15.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:13:15.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:13:15.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 94644 94665 94666 94667 2021-05-12T12:13:15.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2021-05-12T12:13:15.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2021-05-12T12:13:15.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:15.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:13:15.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:13:15.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:13:15.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/0 2021-05-12T12:13:15.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:15.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:15.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:15.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:15.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-05-12T12:13:15.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-05-12T12:13:15.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:15.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:13:15.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:15.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:13:15.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:15.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:15.487 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:15.487 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:15.488 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:13:15.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:13:15.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:13:15.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:15.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:15.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:15.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:15.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:15.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:15.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:13:15.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:13:15.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/0 2021-05-12T12:13:15.490 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:13:15.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:13:15.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:15.492 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/0/whoami 2021-05-12T12:13:15.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:13:15.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:13:15.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:13:15.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:13:15.504 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:15.501+0000 7f86aaa8f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:15.506 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:15.503+0000 7f86aaa8f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:15.508 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:15.505+0000 7f86aaa8f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:15.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:13:15.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:15.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:13:15.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:15.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:15.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:15.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:15.852 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:13:15.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:15.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:13:16.023 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:16.020+0000 7f86aaa8f0c0 -1 Falling back to public interface 2021-05-12T12:13:16.206 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/89799,v1:127.0.0.1:6803/89799] [v2:127.0.0.1:6804/89799,v1:127.0.0.1:6805/89799] exists,up 9e938bc4-7017-45f8-99dd-cea127023b11 2021-05-12T12:13:16.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:16.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:16.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:16.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2021-05-12T12:13:16.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:16.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:13:16.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:13:16.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:13:16.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/1 2021-05-12T12:13:16.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:16.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:16.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:16.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:16.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-05-12T12:13:16.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-05-12T12:13:16.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:16.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:13:16.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:16.214 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:13:16.214 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:16.215 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:16.215 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:16.215 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:16.215 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:13:16.215 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:13:16.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:13:16.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:16.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:16.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:16.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:16.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:16.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:16.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:13:16.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:13:16.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/1 2021-05-12T12:13:16.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:13:16.218 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:13:16.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:16.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/1/whoami 2021-05-12T12:13:16.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:13:16.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:13:16.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:13:16.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:13:16.227 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:16.224+0000 7fb4cfb350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:16.229 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:16.226+0000 7fb4cfb350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:16.231 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:16.228+0000 7fb4cfb350c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:16.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:13:16.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:16.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:13:16.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:16.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:16.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:16.570 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:13:16.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:16.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:16.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:13:16.745 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:16.743+0000 7fb4cfb350c0 -1 Falling back to public interface 2021-05-12T12:13:16.927 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 9 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/91184,v1:127.0.0.1:6811/91184] [v2:127.0.0.1:6812/91184,v1:127.0.0.1:6813/91184] exists,up 42874c9e-3714-454d-93d6-1a7a3e1423dc 2021-05-12T12:13:16.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:16.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:16.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:16.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2021-05-12T12:13:16.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:16.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:13:16.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:13:16.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:13:16.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/2 2021-05-12T12:13:16.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:16.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:16.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:16.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:16.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-05-12T12:13:16.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-05-12T12:13:16.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:16.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:13:16.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:16.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:13:16.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:16.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:16.935 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:16.935 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:16.936 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:13:16.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:13:16.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:13:16.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:16.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:16.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:16.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:16.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:16.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:16.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:13:16.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:13:16.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/2 2021-05-12T12:13:16.938 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:13:16.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:13:16.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:16.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/2/whoami 2021-05-12T12:13:16.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:13:16.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:13:16.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:13:16.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:13:16.948 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:16.945+0000 7f0cb3adb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:16.949 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:16.946+0000 7f0cb3adb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:16.951 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:16.948+0000 7f0cb3adb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:17.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:13:17.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:17.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:13:17.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:17.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:17.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:17.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:17.345 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:13:17.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:17.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:13:17.466 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:17.464+0000 7f0cb3adb0c0 -1 Falling back to public interface 2021-05-12T12:13:17.705 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/92595,v1:127.0.0.1:6819/92595] [v2:127.0.0.1:6820/92595,v1:127.0.0.1:6821/92595] exists,up bf730840-416a-44da-9293-d75232e29834 2021-05-12T12:13:17.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:17.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:17.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:17.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2021-05-12T12:13:17.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:17.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:13:17.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T12:13:17.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:13:17.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/3 2021-05-12T12:13:17.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:17.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:17.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:17.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:17.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-05-12T12:13:17.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-05-12T12:13:17.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:17.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:13:17.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:17.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:13:17.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:17.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:17.714 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:17.714 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:17.714 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:13:17.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:13:17.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' 2021-05-12T12:13:17.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:17.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:17.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:17.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:17.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:17.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:17.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:13:17.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:13:17.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/3 2021-05-12T12:13:17.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T12:13:17.717 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.3 2021-05-12T12:13:17.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=70285401-6582-4327-8855-383bb42cb243 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.88679/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:17.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/3/whoami 2021-05-12T12:13:17.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T12:13:17.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:13:17.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:13:17.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:13:17.726 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:17.723+0000 7f63b1b970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:17.728 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:17.725+0000 7f63b1b970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:17.729 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:17.727+0000 7f63b1b970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:18.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T12:13:18.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:18.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:13:18.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:18.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:18.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:18.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:18.088 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:13:18.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:18.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:13:18.245 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:18.241+0000 7f63b1b970c0 -1 Falling back to public interface 2021-05-12T12:13:18.463 INFO:tasks.workunit.client.0.smithi191.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/94006,v1:127.0.0.1:6827/94006] [v2:127.0.0.1:6828/94006,v1:127.0.0.1:6829/94006] exists,up 8ec65632-58f0-4715-a14d-e33f833d1811 2021-05-12T12:13:18.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:18.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:18.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:18.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2021-05-12T12:13:18.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:13:18.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:13:18.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:13:18.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:13:18.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:13:18.470 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:13:18.470 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:13:18.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:13:18.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:13:18.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:13:18.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:13:18.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:13:18.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:13:18.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:13:18.547 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:13:18.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:13:18.979 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:13:18.979 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:13:18.979 INFO:tasks.workunit.client.0.smithi191.stderr:3' 2021-05-12T12:13:18.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:13:18.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:13:18.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:13:19.303 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:19.300+0000 7f86aaa8f0c0 -1 osd.0 25 log_to_monitors {default=true} 2021-05-12T12:13:20.442 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:20.439+0000 7fb4cfb350c0 -1 osd.1 25 log_to_monitors {default=true} 2021-05-12T12:13:21.240 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:21.237+0000 7f0cb3adb0c0 -1 osd.2 24 log_to_monitors {default=true} 2021-05-12T12:13:21.700 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:21.697+0000 7f63b1b970c0 -1 osd.3 25 log_to_monitors {default=true} 2021-05-12T12:13:24.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=120259084291 2021-05-12T12:13:24.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 120259084291 2021-05-12T12:13:24.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-120259084291' 2021-05-12T12:13:24.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:13:24.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:13:24.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=133143986179 2021-05-12T12:13:24.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 133143986179 2021-05-12T12:13:24.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-120259084291 1-133143986179' 2021-05-12T12:13:24.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:13:24.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:13:24.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=150323855363 2021-05-12T12:13:24.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 150323855363 2021-05-12T12:13:24.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-150323855363' 2021-05-12T12:13:24.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:13:24.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:13:24.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=154618822659 2021-05-12T12:13:24.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 154618822659 2021-05-12T12:13:24.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-150323855363 3-154618822659' 2021-05-12T12:13:24.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:13:24.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-120259084291 2021-05-12T12:13:24.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:13:24.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:13:24.491 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-120259084291 2021-05-12T12:13:24.492 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:13:24.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=120259084291 2021-05-12T12:13:24.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2021-05-12T12:13:24.493 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 120259084291 2021-05-12T12:13:24.493 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:13:24.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 120259084291 -lt 120259084291 2021-05-12T12:13:24.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:13:24.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-133143986179 2021-05-12T12:13:24.890 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:13:24.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:13:24.891 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-133143986179 2021-05-12T12:13:24.891 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:13:24.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=133143986179 2021-05-12T12:13:24.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 133143986179' 2021-05-12T12:13:24.893 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 133143986179 2021-05-12T12:13:24.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:13:25.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 133143986179 -lt 133143986179 2021-05-12T12:13:25.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:13:25.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-150323855363 2021-05-12T12:13:25.258 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:13:25.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:13:25.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-150323855363 2021-05-12T12:13:25.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:13:25.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=150323855363 2021-05-12T12:13:25.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2021-05-12T12:13:25.261 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 150323855363 2021-05-12T12:13:25.261 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:13:25.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 150323855363 -lt 150323855363 2021-05-12T12:13:25.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:13:25.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-154618822659 2021-05-12T12:13:25.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:13:25.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:13:25.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-154618822659 2021-05-12T12:13:25.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:13:25.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=154618822659 2021-05-12T12:13:25.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 154618822659' 2021-05-12T12:13:25.625 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 154618822659 2021-05-12T12:13:25.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:13:26.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 154618822659 -lt 154618822659 2021-05-12T12:13:26.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:13:26.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:13:26.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:13:26.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:13:26.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:13:26.475 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:13:26.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:13:26.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:13:26.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:13:26.476 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:13:26.477 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:13:26.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:13:26.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:13:26.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:13:26.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:13:27.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:13:27.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:13:27.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:13:27.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2021-05-12T12:13:27.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-05-12T12:13:27.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:13:27.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-05-12T12:13:27.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:13:27.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:13:27.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:13:27.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:13:27.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:13:27.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:13:27.405 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:13:27.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:13:27.406 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:13:27.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:13:27.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:13:27.407 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:13:27.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:27.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:13:27.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:13:27.409 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:27.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:13:27.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:13:27.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-05-12T12:13:27.417 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:13:27.417 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:27.417 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:13:27.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.88679 2021-05-12T12:13:27.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:13:27.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:13:27.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:13:27.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/ec-error-rollforward 0 2021-05-12T12:13:27.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-05-12T12:13:27.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:13:27.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-05-12T12:13:27.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:13:27.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:13:27.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:13:27.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:13:27.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:13:27.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:13:27.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:13:27.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:13:27.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:13:27.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:13:27.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:13:27.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:13:27.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:27.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:13:27.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:13:27.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:27.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:13:27.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:13:27.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-05-12T12:13:27.427 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:13:27.428 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:27.428 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.88679 2021-05-12T12:13:27.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.88679 2021-05-12T12:13:27.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:13:27.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:13:27.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:13:27.430 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:13:27.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:13:27.493 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2021-05-12T12:13:27.494 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2021-05-12T12:13:27.554 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:13:27.567 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:13:27.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:13:27.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:13:27.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:13:27.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:13:27.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:13:27.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:13:27.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:13:27.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:13:27.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-bench 2021-05-12T12:13:27.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2021-05-12T12:13:27.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2021-05-12T12:13:27.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2021-05-12T12:13:27.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2021-05-12T12:13:27.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2021-05-12T12:13:27.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2021-05-12T12:13:27.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=bb18ed42-20ff-4a02-a98b-00b625e04dd8 --auth-supported=none ' 2021-05-12T12:13:27.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2021-05-12T12:13:27.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2021-05-12T12:13:27.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2021-05-12T12:13:27.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:13:27.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2021-05-12T12:13:27.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2021-05-12T12:13:27.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2021-05-12T12:13:27.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2021-05-12T12:13:27.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2021-05-12T12:13:27.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-05-12T12:13:27.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:13:27.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-05-12T12:13:27.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:13:27.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:13:27.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:13:27.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:13:27.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:13:27.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:13:27.583 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:13:27.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:13:27.584 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:13:27.584 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:13:27.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:13:27.585 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:13:27.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:27.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:13:27.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:13:27.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:27.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:13:27.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:13:27.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-05-12T12:13:27.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:13:27.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:27.596 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97290 2021-05-12T12:13:27.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97290 2021-05-12T12:13:27.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:13:27.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:13:27.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2021-05-12T12:13:27.598 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:13:27.598 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:27.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97290 2021-05-12T12:13:27.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.97290 2021-05-12T12:13:27.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:13:27.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:13:27.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:13:27.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:13:27.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2021-05-12T12:13:27.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2021-05-12T12:13:27.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2021-05-12T12:13:27.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2021-05-12T12:13:27.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2021-05-12T12:13:27.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:13:27.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:13:27.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:13:27.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2021-05-12T12:13:27.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2021-05-12T12:13:27.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:13:27.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:27.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:27.981 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:27.981 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:27.982 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97290 2021-05-12T12:13:27.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97290/$cluster-$name.asok' 2021-05-12T12:13:27.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.97290/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:13:28.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:13:28.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:13:28.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:13:28.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:13:28.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:13:28.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:13:28.233 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:13:28.234 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:13:28.234 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:13:28.234 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:13:28.234 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:28.235 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97290 2021-05-12T12:13:28.235 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97290/ceph-mon.a.asok 2021-05-12T12:13:28.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:13:28.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.97290/ceph-mon.a.asok config get fsid 2021-05-12T12:13:28.327 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:13:28.327 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:13:28.328 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:13:28.328 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:13:28.328 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:13:28.328 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:13:28.329 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:13:28.329 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:13:28.329 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:13:28.329 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:28.329 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97290 2021-05-12T12:13:28.330 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97290/ceph-mon.a.asok 2021-05-12T12:13:28.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:13:28.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.97290/ceph-mon.a.asok config get mon_host 2021-05-12T12:13:28.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2021-05-12T12:13:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2021-05-12T12:13:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:13:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:13:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:13:28.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2021-05-12T12:13:28.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:13:29.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:13:29.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:29.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:29.292 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:29.292 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:29.293 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97290 2021-05-12T12:13:29.293 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97290/$cluster-$name.asok' 2021-05-12T12:13:29.293 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:13:29.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.97290/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:13:29.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2021-05-12T12:13:29.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2021-05-12T12:13:29.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:13:29.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:13:29.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:13:29.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2021-05-12T12:13:29.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=bb18ed42-20ff-4a02-a98b-00b625e04dd8 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2021-05-12T12:13:29.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:29.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:29.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:29.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2021-05-12T12:13:29.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2021-05-12T12:13:29.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:29.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:13:29.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2021-05-12T12:13:29.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:13:29.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:29.321 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:29.321 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:29.321 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:29.321 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97290 2021-05-12T12:13:29.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97290/$cluster-$name.asok' 2021-05-12T12:13:29.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97290/$cluster-$name.asok' 2021-05-12T12:13:29.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:29.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:13:29.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:13:29.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2021-05-12T12:13:29.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2021-05-12T12:13:29.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:29.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:29.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:29.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:13:29.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:13:29.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bench/0 2021-05-12T12:13:29.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:13:29.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=520aca48-3eb3-42a2-be61-d78458afbab5 2021-05-12T12:13:29.325 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 520aca48-3eb3-42a2-be61-d78458afbab5 2021-05-12T12:13:29.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 520aca48-3eb3-42a2-be61-d78458afbab5' 2021-05-12T12:13:29.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:13:29.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBpxptg0jyAExAAJ3O7WOkhd+yags5HTqxsDg== 2021-05-12T12:13:29.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBpxptg0jyAExAAJ3O7WOkhd+yags5HTqxsDg=="}' 2021-05-12T12:13:29.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 520aca48-3eb3-42a2-be61-d78458afbab5 -i td/osd-bench/0/new.json 2021-05-12T12:13:29.703 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:13:29.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bench/0/new.json 2021-05-12T12:13:29.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=bb18ed42-20ff-4a02-a98b-00b625e04dd8 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.97290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBpxptg0jyAExAAJ3O7WOkhd+yags5HTqxsDg== --osd-uuid 520aca48-3eb3-42a2-be61-d78458afbab5 2021-05-12T12:13:29.752 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:29.748+0000 7fb6b75680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:29.763 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:29.760+0000 7fb6b75680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:29.767 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:29.761+0000 7fb6b75680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:29.767 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:29.761+0000 7fb6b75680c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-05-12T12:13:29.768 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:29.761+0000 7fb6b75680c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-05-12T12:13:29.768 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:29.761+0000 7fb6b75680c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-05-12T12:13:29.821 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:29.818+0000 7fb6b75680c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2021-05-12T12:13:34.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bench/0/keyring 2021-05-12T12:13:34.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:13:34.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:13:34.923 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:13:34.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:13:35.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:13:35.412 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:13:35.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=bb18ed42-20ff-4a02-a98b-00b625e04dd8 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.97290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:35.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:13:35.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:13:35.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:13:35.431 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:35.428+0000 7f25ba4b40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:35.433 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:35.430+0000 7f25ba4b40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:35.434 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:35.431+0000 7f25ba4b40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:35.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:13:35.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:35.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:13:35.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:35.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:35.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:35.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:35.768 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:13:35.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:35.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:13:35.958 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:35.955+0000 7f25ba4b40c0 -1 Falling back to public interface 2021-05-12T12:13:36.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:37.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:37.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:37.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:13:37.126 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:13:37.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:37.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:13:37.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:38.231 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:13:38.228+0000 7f25ba4b40c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:13:38.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:38.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:38.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:13:38.492 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:13:38.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:38.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:13:38.886 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/98410,v1:127.0.0.1:6803/98410] [v2:127.0.0.1:6804/98410,v1:127.0.0.1:6805/98410] exists,up 520aca48-3eb3-42a2-be61-d78458afbab5 2021-05-12T12:13:38.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:38.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:38.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:38.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2021-05-12T12:13:38.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2021-05-12T12:13:38.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2021-05-12T12:13:38.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2021-05-12T12:13:38.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2021-05-12T12:13:38.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2021-05-12T12:13:38.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2021-05-12T12:13:38.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2021-05-12T12:13:38.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2021-05-12T12:13:38.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2021-05-12T12:13:38.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2021-05-12T12:13:38.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2021-05-12T12:13:38.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2021-05-12T12:13:38.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: local dir=td/osd-bench 2021-05-12T12:13:38.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: shift 2021-05-12T12:13:38.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: local expected=osd_bench_max_block_size 2021-05-12T12:13:38.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: shift 2021-05-12T12:13:38.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: local success 2021-05-12T12:13:38.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: ceph tell osd.0 bench 1024 67108865 2021-05-12T12:13:39.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: success=false 2021-05-12T12:13:39.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: false 2021-05-12T12:13:39.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2021-05-12T12:13:39.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: expect_failure: return 0 2021-05-12T12:13:39.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2021-05-12T12:13:39.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2021-05-12T12:13:39.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2021-05-12T12:13:39.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: local dir=td/osd-bench 2021-05-12T12:13:39.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: shift 2021-05-12T12:13:39.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: local expected=bench_small_size_max_iops 2021-05-12T12:13:39.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: shift 2021-05-12T12:13:39.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: local success 2021-05-12T12:13:39.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: ceph tell osd.0 bench 3072001 1024 2021-05-12T12:13:39.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: success=false 2021-05-12T12:13:39.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: false 2021-05-12T12:13:39.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2021-05-12T12:13:39.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: expect_failure: return 0 2021-05-12T12:13:39.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2021-05-12T12:13:39.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2021-05-12T12:13:39.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2021-05-12T12:13:39.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: local dir=td/osd-bench 2021-05-12T12:13:39.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: shift 2021-05-12T12:13:39.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: local expected=osd_bench_large_size_max_throughput 2021-05-12T12:13:39.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: shift 2021-05-12T12:13:39.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: local success 2021-05-12T12:13:39.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2021-05-12T12:13:39.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: success=false 2021-05-12T12:13:39.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: false 2021-05-12T12:13:39.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2021-05-12T12:13:39.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: expect_failure: return 0 2021-05-12T12:13:39.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2021-05-12T12:14:51.440 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:14:51.441 INFO:tasks.workunit.client.0.smithi191.stdout: "bytes_written": 1073741824, 2021-05-12T12:14:51.441 INFO:tasks.workunit.client.0.smithi191.stdout: "blocksize": 4194304, 2021-05-12T12:14:51.441 INFO:tasks.workunit.client.0.smithi191.stdout: "elapsed_sec": 37.449091224999997, 2021-05-12T12:14:51.442 INFO:tasks.workunit.client.0.smithi191.stdout: "bytes_per_sec": 28672039.530913878, 2021-05-12T12:14:51.442 INFO:tasks.workunit.client.0.smithi191.stdout: "iops": 6.8359469249043174 2021-05-12T12:14:51.442 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:14:51.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2021-05-12T12:14:52.238 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:14:52.239 INFO:tasks.workunit.client.0.smithi191.stdout: "bytes_written": 10, 2021-05-12T12:14:52.239 INFO:tasks.workunit.client.0.smithi191.stdout: "blocksize": 4444, 2021-05-12T12:14:52.239 INFO:tasks.workunit.client.0.smithi191.stdout: "elapsed_sec": 0.058653757000000001, 2021-05-12T12:14:52.240 INFO:tasks.workunit.client.0.smithi191.stdout: "bytes_per_sec": 170.49206242662342, 2021-05-12T12:14:52.240 INFO:tasks.workunit.client.0.smithi191.stdout: "iops": 0.038364550501040373 2021-05-12T12:14:52.240 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:14:52.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2021-05-12T12:14:53.003 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:14:53.003 INFO:tasks.workunit.client.0.smithi191.stdout: "bytes_written": 1, 2021-05-12T12:14:53.003 INFO:tasks.workunit.client.0.smithi191.stdout: "blocksize": 14456, 2021-05-12T12:14:53.004 INFO:tasks.workunit.client.0.smithi191.stdout: "elapsed_sec": 0.40376605599999998, 2021-05-12T12:14:53.004 INFO:tasks.workunit.client.0.smithi191.stdout: "bytes_per_sec": 2.4766816951051478, 2021-05-12T12:14:53.004 INFO:tasks.workunit.client.0.smithi191.stdout: "iops": 0.00017132551847711316 2021-05-12T12:14:53.004 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:14:53.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2021-05-12T12:14:53.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-05-12T12:14:53.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:14:53.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-05-12T12:14:53.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:14:53.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:14:53.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:14:53.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:14:53.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:14:53.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:14:53.125 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:14:53.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:14:53.126 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:14:53.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:14:53.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:14:53.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:14:53.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:14:53.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:14:53.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:14:53.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:14:53.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:14:53.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:14:53.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-05-12T12:14:53.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:14:53.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:14:53.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97290 2021-05-12T12:14:53.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97290 2021-05-12T12:14:53.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:14:53.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:14:53.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:14:53.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-bench 0 2021-05-12T12:14:53.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-05-12T12:14:53.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:14:53.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-05-12T12:14:53.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:14:53.535 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:14:53.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:14:53.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:14:53.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:14:53.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:14:53.537 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:14:53.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:14:53.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:14:53.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:14:53.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:14:53.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:14:53.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:14:53.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:14:53.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:14:53.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:14:53.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:14:53.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:14:53.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-05-12T12:14:53.541 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:14:53.541 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:14:53.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97290 2021-05-12T12:14:53.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97290 2021-05-12T12:14:53.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:14:53.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:14:53.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:14:53.544 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:14:53.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:14:53.608 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2021-05-12T12:14:53.608 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2021-05-12T12:14:53.668 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:14:53.677 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:14:53.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:14:53.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:14:53.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:14:53.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:14:53.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:14:53.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:14:53.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:14:53.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:14:53.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-bluefs-volume-ops 2021-05-12T12:14:53.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:14:53.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2021-05-12T12:14:53.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-05-12T12:14:53.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-05-12T12:14:53.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: export CEPH_ARGS 2021-05-12T12:14:53.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: uuidgen 2021-05-12T12:14:53.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: CEPH_ARGS+='--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none ' 2021-05-12T12:14:53.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-05-12T12:14:53.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2021-05-12T12:14:53.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:16: run: CEPH_ARGS+='--bluestore_block_db_create=true ' 2021-05-12T12:14:53.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:17: run: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2021-05-12T12:14:53.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:18: run: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2021-05-12T12:14:53.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:19: run: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2021-05-12T12:14:53.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: run: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2021-05-12T12:14:53.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: set 2021-05-12T12:14:53.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:14:53.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: local funcs=TEST_bluestore 2021-05-12T12:14:53.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: run: for func in $funcs 2021-05-12T12:14:53.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: run: setup td/osd-bluefs-volume-ops 2021-05-12T12:14:53.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:14:53.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2021-05-12T12:14:53.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:14:53.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:14:53.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-05-12T12:14:53.686 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:14:53.687 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:14:53.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:14:53.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:14:53.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:14:53.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:14:53.687 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:14:53.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:14:53.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:14:53.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:14:53.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:14:53.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:14:53.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:14:53.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:14:53.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:14:53.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:14:53.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:14:53.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:14:53.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-05-12T12:14:53.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:14:53.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:14:53.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:14:53.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.99184 2021-05-12T12:14:53.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:14:53.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:14:53.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2021-05-12T12:14:53.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:14:53.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:14:53.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:14:53.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.99184 2021-05-12T12:14:53.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:14:53.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:14:53.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:14:53.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:14:53.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2021-05-12T12:14:53.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:24: run: TEST_bluestore td/osd-bluefs-volume-ops 2021-05-12T12:14:53.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:14:53.697 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: ulimit -n 2021-05-12T12:14:53.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: local flimit=4096 2021-05-12T12:14:53.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: '[' 4096 -lt 1536 ']' 2021-05-12T12:14:53.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2021-05-12T12:14:53.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:14:53.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:14:53.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:14:53.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:14:53.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2021-05-12T12:14:53.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2021-05-12T12:14:54.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:14:54.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:14:54.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:14:54.401 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:14:54.401 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:14:54.402 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:14:54.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:14:54.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:14:54.654 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:14:54.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:14:54.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:14:54.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:14:54.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:14:54.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:14:54.658 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:14:54.658 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:14:54.658 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:14:54.658 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:14:54.658 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:14:54.659 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:14:54.659 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.99184/ceph-mon.a.asok 2021-05-12T12:14:54.659 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:14:54.659 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.99184/ceph-mon.a.asok config get fsid 2021-05-12T12:14:54.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:14:54.757 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:14:54.757 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:14:54.757 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:14:54.757 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:14:54.758 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:14:54.758 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:14:54.758 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:14:54.758 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:14:54.759 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:14:54.759 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:14:54.759 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.99184/ceph-mon.a.asok 2021-05-12T12:14:54.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:14:54.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.99184/ceph-mon.a.asok config get mon_host 2021-05-12T12:14:54.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2021-05-12T12:14:54.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:14:54.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:14:54.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:14:54.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:14:54.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2021-05-12T12:14:54.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:14:55.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:14:55.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:14:55.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:14:55.760 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:14:55.760 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:14:55.761 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:14:55.761 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:14:55.761 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:14:55.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:14:55.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2021-05-12T12:14:55.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:14:55.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:14:55.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:14:55.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:14:55.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T12:14:55.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:14:55.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:14:55.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:14:55.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:14:55.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T12:14:55.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T12:14:55.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:14:55.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:14:55.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:14:55.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:14:55.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:14:55.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:14:55.789 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:14:55.789 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:14:55.790 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:14:55.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:14:55.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:14:55.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:14:55.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:14:55.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:14:55.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:14:55.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:14:55.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:14:55.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:14:55.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:14:55.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:14:55.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:14:55.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T12:14:55.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:14:55.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=d2d373bf-6a2c-4915-afa2-7895259f1e1d 2021-05-12T12:14:55.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 d2d373bf-6a2c-4915-afa2-7895259f1e1d' 2021-05-12T12:14:55.793 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 d2d373bf-6a2c-4915-afa2-7895259f1e1d 2021-05-12T12:14:55.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:14:55.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC/xptgMadsLxAA2oIg6A/JxLz+6cUiVI+Ukw== 2021-05-12T12:14:55.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC/xptgMadsLxAA2oIg6A/JxLz+6cUiVI+Ukw=="}' 2021-05-12T12:14:55.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new d2d373bf-6a2c-4915-afa2-7895259f1e1d -i td/osd-bluefs-volume-ops/0/new.json 2021-05-12T12:14:56.137 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:14:56.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2021-05-12T12:14:56.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC/xptgMadsLxAA2oIg6A/JxLz+6cUiVI+Ukw== --osd-uuid d2d373bf-6a2c-4915-afa2-7895259f1e1d 2021-05-12T12:14:56.170 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:14:56.167+0000 7f9ff937a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:14:56.172 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:14:56.169+0000 7f9ff937a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:14:56.174 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:14:56.170+0000 7f9ff937a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:14:56.175 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:14:56.171+0000 7f9ff937a0c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-05-12T12:14:56.175 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:14:56.171+0000 7f9ff937a0c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-05-12T12:14:56.175 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:14:56.171+0000 7f9ff937a0c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-05-12T12:14:56.211 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:14:56.208+0000 7f9ff937a0c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2021-05-12T12:15:04.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2021-05-12T12:15:04.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:15:04.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:15:04.869 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:15:04.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:15:05.352 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:15:05.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:15:05.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:05.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:15:05.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:15:05.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:15:05.372 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:05.369+0000 7fb2ea99a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:05.374 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:05.371+0000 7fb2ea99a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:05.375 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:05.372+0000 7fb2ea99a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:05.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:15:05.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:15:05.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:15:05.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:15:05.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:15:05.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:05.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:15:05.713 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:15:05.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:05.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:06.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:06.391 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:06.388+0000 7fb2ea99a0c0 -1 Falling back to public interface 2021-05-12T12:15:07.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:07.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:07.068 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:15:07.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:15:07.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:07.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:07.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:08.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:08.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:08.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:15:08.467 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:15:08.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:08.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:08.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:09.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:09.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:09.828 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:15:09.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:15:09.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:09.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:10.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:10.978 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:10.975+0000 7fb2ea99a0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:15:11.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:11.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:11.193 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:15:11.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:15:11.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:11.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:11.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:12.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:12.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:12.610 INFO:tasks.workunit.client.0.smithi191.stdout:5 2021-05-12T12:15:12.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 5 2021-05-12T12:15:12.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:12.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:12.983 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/100701,v1:127.0.0.1:6803/100701] [v2:127.0.0.1:6804/100701,v1:127.0.0.1:6805/100701] exists,up d2d373bf-6a2c-4915-afa2-7895259f1e1d 2021-05-12T12:15:12.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:15:12.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:15:12.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:15:12.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T12:15:12.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=100701 2021-05-12T12:15:12.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2021-05-12T12:15:12.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:12.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:15:12.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:15:12.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:15:12.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T12:15:12.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:15:12.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:15:12.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:15:12.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:15:12.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T12:15:12.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T12:15:12.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:15:12.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:15:12.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:15:12.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:15:12.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:15:12.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:15:12.993 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:15:12.993 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:12.993 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:15:12.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:15:12.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:15:12.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:15:12.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:15:12.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:15:12.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:15:12.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:15:12.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:15:12.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:15:12.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:12.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:15:12.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:15:12.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T12:15:12.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:15:12.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f71f7391-473d-4cc3-8e29-43a620de60a1 2021-05-12T12:15:12.997 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 f71f7391-473d-4cc3-8e29-43a620de60a1 2021-05-12T12:15:12.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 f71f7391-473d-4cc3-8e29-43a620de60a1' 2021-05-12T12:15:12.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:15:13.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDQxptgFFSKOxAAZftUmxj/mcrR/kf9ocnPyA== 2021-05-12T12:15:13.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDQxptgFFSKOxAAZftUmxj/mcrR/kf9ocnPyA=="}' 2021-05-12T12:15:13.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f71f7391-473d-4cc3-8e29-43a620de60a1 -i td/osd-bluefs-volume-ops/1/new.json 2021-05-12T12:15:13.461 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:15:13.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2021-05-12T12:15:13.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDQxptgFFSKOxAAZftUmxj/mcrR/kf9ocnPyA== --osd-uuid f71f7391-473d-4cc3-8e29-43a620de60a1 2021-05-12T12:15:13.494 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:13.491+0000 7f05a26510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:13.496 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:13.493+0000 7f05a26510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:13.498 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:13.495+0000 7f05a26510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:13.499 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:13.496+0000 7f05a26510c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-05-12T12:15:13.499 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:13.496+0000 7f05a26510c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-05-12T12:15:13.499 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:13.496+0000 7f05a26510c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-05-12T12:15:13.532 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:13.530+0000 7f05a26510c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2021-05-12T12:15:22.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2021-05-12T12:15:22.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:15:22.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:15:22.617 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:15:22.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:15:23.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:15:23.101 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:15:23.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:23.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:15:23.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:15:23.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:15:23.121 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:23.118+0000 7fd002f190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:23.122 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:23.120+0000 7fd002f190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:23.123 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:23.120+0000 7fd002f190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:23.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:15:23.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:15:23.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:15:23.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:15:23.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:15:23.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:23.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:15:23.498 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:15:23.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:23.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:23.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:24.140 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:24.137+0000 7fd002f190c0 -1 Falling back to public interface 2021-05-12T12:15:24.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:24.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:24.853 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:15:24.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:15:24.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:24.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:25.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:26.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:26.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:26.218 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:15:26.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:15:26.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:26.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:26.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:27.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:27.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:27.591 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:15:27.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:15:27.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:27.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:27.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:28.686 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:28.683+0000 7fd002f190c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:15:28.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:28.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:28.958 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:15:28.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:15:28.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:28.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:29.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:30.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:30.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:30.325 INFO:tasks.workunit.client.0.smithi191.stdout:5 2021-05-12T12:15:30.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 5 2021-05-12T12:15:30.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:30.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:30.696 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/102947,v1:127.0.0.1:6811/102947] [v2:127.0.0.1:6812/102947,v1:127.0.0.1:6813/102947] exists,up f71f7391-473d-4cc3-8e29-43a620de60a1 2021-05-12T12:15:30.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:15:30.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:15:30.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:15:30.697 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T12:15:30.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=102947 2021-05-12T12:15:30.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2021-05-12T12:15:30.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:30.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:15:30.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:15:30.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:15:30.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T12:15:30.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:15:30.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:15:30.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:15:30.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:15:30.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T12:15:30.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T12:15:30.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:15:30.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:15:30.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:15:30.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:15:30.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:15:30.707 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:15:30.707 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:15:30.707 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:30.707 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:15:30.708 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:15:30.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:15:30.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:15:30.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:15:30.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:15:30.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:15:30.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:15:30.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:15:30.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:15:30.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:30.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:15:30.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:15:30.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T12:15:30.711 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:15:30.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=ad18b5ee-7ebd-47ea-8856-c1d530f45233 2021-05-12T12:15:30.711 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 ad18b5ee-7ebd-47ea-8856-c1d530f45233 2021-05-12T12:15:30.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 ad18b5ee-7ebd-47ea-8856-c1d530f45233' 2021-05-12T12:15:30.712 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:15:30.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDixptgrxx/KhAAbAj2BDOFzqn8+RTM35IzNQ== 2021-05-12T12:15:30.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDixptgrxx/KhAAbAj2BDOFzqn8+RTM35IzNQ=="}' 2021-05-12T12:15:30.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new ad18b5ee-7ebd-47ea-8856-c1d530f45233 -i td/osd-bluefs-volume-ops/2/new.json 2021-05-12T12:15:31.235 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:15:31.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2021-05-12T12:15:31.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDixptgrxx/KhAAbAj2BDOFzqn8+RTM35IzNQ== --osd-uuid ad18b5ee-7ebd-47ea-8856-c1d530f45233 2021-05-12T12:15:31.268 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:31.265+0000 7f49053c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:31.270 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:31.267+0000 7f49053c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:31.272 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:31.268+0000 7f49053c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:31.273 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:31.270+0000 7f49053c50c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-05-12T12:15:31.273 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:31.270+0000 7f49053c50c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-05-12T12:15:31.273 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:31.270+0000 7f49053c50c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-05-12T12:15:31.312 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:31.309+0000 7f49053c50c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2021-05-12T12:15:40.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2021-05-12T12:15:40.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:15:40.463 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:15:40.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:15:40.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:15:40.942 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:15:40.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:15:40.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:40.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:15:40.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:15:40.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:15:40.962 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:40.959+0000 7fb3bdb730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:40.964 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:40.961+0000 7fb3bdb730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:40.965 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:40.962+0000 7fb3bdb730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:41.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:15:41.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:15:41.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:15:41.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:15:41.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:15:41.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:41.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:15:41.303 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:15:41.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:41.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:41.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:41.982 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:41.979+0000 7fb3bdb730c0 -1 Falling back to public interface 2021-05-12T12:15:42.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:42.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:42.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:15:42.660 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:15:42.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:42.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:43.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:44.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:44.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:44.050 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:15:44.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:15:44.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:44.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:44.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:45.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:45.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:45.419 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:15:45.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:15:45.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:45.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:45.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:46.591 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:46.588+0000 7fb3bdb730c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:15:46.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:46.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:46.778 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:15:46.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:15:46.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:46.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:47.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:48.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:48.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:48.149 INFO:tasks.workunit.client.0.smithi191.stdout:5 2021-05-12T12:15:48.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 5 2021-05-12T12:15:48.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:48.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:48.521 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/105201,v1:127.0.0.1:6819/105201] [v2:127.0.0.1:6820/105201,v1:127.0.0.1:6821/105201] exists,up ad18b5ee-7ebd-47ea-8856-c1d530f45233 2021-05-12T12:15:48.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:15:48.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:15:48.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:15:48.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T12:15:48.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=105201 2021-05-12T12:15:48.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2021-05-12T12:15:48.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:48.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:15:48.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-05-12T12:15:48.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:15:48.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T12:15:48.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:15:48.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:15:48.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:15:48.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:15:48.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T12:15:48.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T12:15:48.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:15:48.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:15:48.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:15:48.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:15:48.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:15:48.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:15:48.532 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:15:48.532 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:48.532 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:15:48.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:15:48.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:15:48.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:15:48.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:15:48.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:15:48.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:15:48.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:15:48.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:15:48.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:15:48.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:48.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:15:48.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:15:48.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T12:15:48.535 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:15:48.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=16258e95-8742-4373-96ca-e9d8bda153b0 2021-05-12T12:15:48.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 16258e95-8742-4373-96ca-e9d8bda153b0' 2021-05-12T12:15:48.536 INFO:tasks.workunit.client.0.smithi191.stdout:add osd3 16258e95-8742-4373-96ca-e9d8bda153b0 2021-05-12T12:15:48.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:15:48.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD0xptgK5UUIBAA/DOfYSYSdYs/DTc+soQNMw== 2021-05-12T12:15:48.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD0xptgK5UUIBAA/DOfYSYSdYs/DTc+soQNMw=="}' 2021-05-12T12:15:48.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 16258e95-8742-4373-96ca-e9d8bda153b0 -i td/osd-bluefs-volume-ops/3/new.json 2021-05-12T12:15:49.043 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:15:49.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2021-05-12T12:15:49.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD0xptgK5UUIBAA/DOfYSYSdYs/DTc+soQNMw== --osd-uuid 16258e95-8742-4373-96ca-e9d8bda153b0 2021-05-12T12:15:49.076 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:49.073+0000 7f320956e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:49.078 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:49.075+0000 7f320956e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:49.080 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:49.077+0000 7f320956e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:49.081 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:49.078+0000 7f320956e0c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-05-12T12:15:49.081 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:49.078+0000 7f320956e0c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-05-12T12:15:49.082 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:49.078+0000 7f320956e0c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-05-12T12:15:49.120 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:49.118+0000 7f320956e0c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2021-05-12T12:15:58.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2021-05-12T12:15:58.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:15:58.102 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd3 key to auth repository 2021-05-12T12:15:58.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-05-12T12:15:58.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:15:58.589 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.3 2021-05-12T12:15:58.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-05-12T12:15:58.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:15:58.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:58.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:15:58.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:15:58.608 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:58.605+0000 7fde044b60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:58.610 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:58.607+0000 7fde044b60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:58.611 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:58.608+0000 7fde044b60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:58.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-05-12T12:15:58.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:15:58.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:15:58.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:15:58.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:15:58.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:58.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:15:58.978 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:15:58.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:58.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:15:59.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:59.627 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:15:59.624+0000 7fde044b60c0 -1 Falling back to public interface 2021-05-12T12:16:00.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:16:00.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:16:00.332 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:16:00.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:16:00.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:16:00.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:16:00.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:16:01.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:16:01.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:16:01.725 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:16:01.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:16:01.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:16:01.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:16:02.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:16:03.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:16:03.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:16:03.096 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:16:03.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:16:03.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:16:03.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:16:03.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:16:03.929 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:16:03.926+0000 7fde044b60c0 -1 osd.3 0 log_to_monitors {default=true} 2021-05-12T12:16:04.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:16:04.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:16:04.467 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:16:04.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:16:04.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:16:04.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:16:04.853 INFO:tasks.workunit.client.0.smithi191.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/107447,v1:127.0.0.1:6827/107447] [v2:127.0.0.1:6828/107447,v1:127.0.0.1:6829/107447] exists,up 16258e95-8742-4373-96ca-e9d8bda153b0 2021-05-12T12:16:04.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:16:04.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:16:04.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:16:04.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T12:16:04.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=107447 2021-05-12T12:16:04.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2021-05-12T12:16:09.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2021-05-12T12:16:09.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2021-05-12T12:16:10.689 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foo' already exists 2021-05-12T12:16:10.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:16:11.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T12:16:11.749 INFO:tasks.workunit.client.0.smithi191.stdout:hints = 1 2021-05-12T12:16:11.749 INFO:tasks.workunit.client.0.smithi191.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T12:16:11.749 INFO:tasks.workunit.client.0.smithi191.stdout:Object prefix: benchmark_data_smithi191_108457 2021-05-12T12:16:11.750 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:16:11.750 INFO:tasks.workunit.client.0.smithi191.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:16:12.749 INFO:tasks.workunit.client.0.smithi191.stdout: 1 16 48 32 0.124986 0.125 0.257499 0.4183 2021-05-12T12:16:13.749 INFO:tasks.workunit.client.0.smithi191.stdout: 2 16 87 71 0.138654 0.152344 0.243532 0.401959 2021-05-12T12:16:14.749 INFO:tasks.workunit.client.0.smithi191.stdout: 3 16 141 125 0.162741 0.210938 0.301029 0.359421 2021-05-12T12:16:15.749 INFO:tasks.workunit.client.0.smithi191.stdout: 4 16 198 182 0.177713 0.222656 0.29343 0.34481 2021-05-12T12:16:16.749 INFO:tasks.workunit.client.0.smithi191.stdout: 5 16 251 235 0.183571 0.207031 0.309897 0.331511 2021-05-12T12:16:17.750 INFO:tasks.workunit.client.0.smithi191.stdout: 6 16 301 285 0.185524 0.195312 0.318097 0.3285 2021-05-12T12:16:18.750 INFO:tasks.workunit.client.0.smithi191.stdout: 7 16 349 333 0.185802 0.1875 0.302819 0.32785 2021-05-12T12:16:19.750 INFO:tasks.workunit.client.0.smithi191.stdout: 8 16 397 381 0.186011 0.1875 0.301859 0.326056 2021-05-12T12:16:20.751 INFO:tasks.workunit.client.0.smithi191.stdout: 9 16 454 438 0.19008 0.222656 0.336889 0.324887 2021-05-12T12:16:21.750 INFO:tasks.workunit.client.0.smithi191.stdout: 10 16 502 486 0.189819 0.1875 0.267366 0.3219 2021-05-12T12:16:22.750 INFO:tasks.workunit.client.0.smithi191.stdout: 11 16 557 541 0.192091 0.214844 0.318386 0.320079 2021-05-12T12:16:23.750 INFO:tasks.workunit.client.0.smithi191.stdout: 12 16 605 589 0.191707 0.1875 0.328244 0.320939 2021-05-12T12:16:24.751 INFO:tasks.workunit.client.0.smithi191.stdout: 13 16 653 637 0.191381 0.1875 0.359636 0.320717 2021-05-12T12:16:25.751 INFO:tasks.workunit.client.0.smithi191.stdout: 14 16 701 685 0.191102 0.1875 0.342703 0.322201 2021-05-12T12:16:26.751 INFO:tasks.workunit.client.0.smithi191.stdout: 15 16 758 742 0.193204 0.222656 0.325026 0.320264 2021-05-12T12:16:27.751 INFO:tasks.workunit.client.0.smithi191.stdout: 16 16 806 790 0.192847 0.1875 0.241404 0.320084 2021-05-12T12:16:28.751 INFO:tasks.workunit.client.0.smithi191.stdout: 17 16 854 838 0.192531 0.1875 0.43586 0.321018 2021-05-12T12:16:29.751 INFO:tasks.workunit.client.0.smithi191.stdout: 18 16 909 893 0.193769 0.214844 0.268683 0.319202 2021-05-12T12:16:30.751 INFO:tasks.workunit.client.0.smithi191.stdout: 19 16 966 950 0.195288 0.222656 0.314383 0.31796 2021-05-12T12:16:31.751 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T12:16:31.749713+0000 min lat: 0.175907 max lat: 0.636223 avg lat: 0.316022 2021-05-12T12:16:31.752 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:16:31.752 INFO:tasks.workunit.client.0.smithi191.stdout: 20 16 1021 1005 0.196264 0.214844 0.293568 0.316022 2021-05-12T12:16:32.751 INFO:tasks.workunit.client.0.smithi191.stdout: 21 16 1069 1053 0.195845 0.1875 0.309944 0.315624 2021-05-12T12:16:33.752 INFO:tasks.workunit.client.0.smithi191.stdout: 22 16 1117 1101 0.195465 0.1875 0.408403 0.316145 2021-05-12T12:16:34.752 INFO:tasks.workunit.client.0.smithi191.stdout: 23 16 1174 1158 0.196646 0.222656 0.310945 0.316366 2021-05-12T12:16:35.752 INFO:tasks.workunit.client.0.smithi191.stdout: 24 16 1222 1206 0.196265 0.1875 0.336754 0.316382 2021-05-12T12:16:36.752 INFO:tasks.workunit.client.0.smithi191.stdout: 25 16 1277 1261 0.197007 0.214844 0.309044 0.315551 2021-05-12T12:16:37.752 INFO:tasks.workunit.client.0.smithi191.stdout: 26 16 1325 1309 0.19664 0.1875 0.303097 0.315605 2021-05-12T12:16:38.752 INFO:tasks.workunit.client.0.smithi191.stdout: 27 16 1382 1366 0.197603 0.222656 0.302875 0.315049 2021-05-12T12:16:39.752 INFO:tasks.workunit.client.0.smithi191.stdout: 28 16 1437 1421 0.198217 0.214844 0.237847 0.314263 2021-05-12T12:16:40.752 INFO:tasks.workunit.client.0.smithi191.stdout: 29 16 1494 1478 0.199059 0.222656 0.254927 0.31277 2021-05-12T12:16:41.753 INFO:tasks.workunit.client.0.smithi191.stdout: 30 16 1542 1526 0.198673 0.1875 0.261443 0.312351 2021-05-12T12:16:42.753 INFO:tasks.workunit.client.0.smithi191.stdout:Total time run: 30.1125 2021-05-12T12:16:42.753 INFO:tasks.workunit.client.0.smithi191.stdout:Total writes made: 1542 2021-05-12T12:16:42.753 INFO:tasks.workunit.client.0.smithi191.stdout:Write size: 4096 2021-05-12T12:16:42.754 INFO:tasks.workunit.client.0.smithi191.stdout:Object size: 4096 2021-05-12T12:16:42.754 INFO:tasks.workunit.client.0.smithi191.stdout:Bandwidth (MB/sec): 0.200031 2021-05-12T12:16:42.754 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Bandwidth: 0.0225621 2021-05-12T12:16:42.754 INFO:tasks.workunit.client.0.smithi191.stdout:Max bandwidth (MB/sec): 0.222656 2021-05-12T12:16:42.755 INFO:tasks.workunit.client.0.smithi191.stdout:Min bandwidth (MB/sec): 0.125 2021-05-12T12:16:42.755 INFO:tasks.workunit.client.0.smithi191.stdout:Average IOPS: 51 2021-05-12T12:16:42.755 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev IOPS: 5.77589 2021-05-12T12:16:42.755 INFO:tasks.workunit.client.0.smithi191.stdout:Max IOPS: 57 2021-05-12T12:16:42.755 INFO:tasks.workunit.client.0.smithi191.stdout:Min IOPS: 32 2021-05-12T12:16:42.756 INFO:tasks.workunit.client.0.smithi191.stdout:Average Latency(s): 0.311985 2021-05-12T12:16:42.756 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Latency(s): 0.0493353 2021-05-12T12:16:42.756 INFO:tasks.workunit.client.0.smithi191.stdout:Max latency(s): 0.636223 2021-05-12T12:16:42.756 INFO:tasks.workunit.client.0.smithi191.stdout:Min latency(s): 0.175907 2021-05-12T12:16:42.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2021-05-12T12:16:42.832 INFO:tasks.workunit.client.0.smithi191.stdout:after bench 2021-05-12T12:16:42.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 100701 2021-05-12T12:16:42.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2021-05-12T12:16:43.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 100701 2021-05-12T12:16:43.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (100701) - No such process 2021-05-12T12:16:43.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2021-05-12T12:16:44.232 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already down. 2021-05-12T12:16:44.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 102947 2021-05-12T12:16:44.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2021-05-12T12:16:45.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 102947 2021-05-12T12:16:45.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (102947) - No such process 2021-05-12T12:16:45.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2021-05-12T12:16:45.894 INFO:tasks.workunit.client.0.smithi191.stderr:osd.1 is already down. 2021-05-12T12:16:45.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 105201 2021-05-12T12:16:45.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2021-05-12T12:16:46.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 105201 2021-05-12T12:16:46.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (105201) - No such process 2021-05-12T12:16:46.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2021-05-12T12:16:47.290 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already down. 2021-05-12T12:16:47.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 107447 2021-05-12T12:16:47.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2021-05-12T12:16:48.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 107447 2021-05-12T12:16:48.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (107447) - No such process 2021-05-12T12:16:48.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2021-05-12T12:16:48.795 INFO:tasks.workunit.client.0.smithi191.stderr:osd.3 is already down. 2021-05-12T12:16:48.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:16:50.585 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:16:50.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:16:52.375 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:16:52.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:16:54.163 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:16:54.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:16:55.952 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:16:55.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2021-05-12T12:16:55.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2021-05-12T12:16:55.973 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:16:56.759 INFO:tasks.workunit.client.0.smithi191.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-05-12T12:16:56.759 INFO:tasks.workunit.client.0.smithi191.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T12:16:56.760 INFO:tasks.workunit.client.0.smithi191.stdout:2 : device size 0x100000000 : using 0x804a8000(2.0 GiB) 2021-05-12T12:16:56.760 INFO:tasks.workunit.client.0.smithi191.stdout:Expanding DB/WAL... 2021-05-12T12:16:56.760 INFO:tasks.workunit.client.0.smithi191.stdout:0 : expanding to 0x536870912 2021-05-12T12:16:56.819 INFO:tasks.workunit.client.0.smithi191.stdout:0 : size label updated to 536870912 2021-05-12T12:16:56.820 INFO:tasks.workunit.client.0.smithi191.stdout:1 : expanding to 0x1073741824 2021-05-12T12:16:56.848 INFO:tasks.workunit.client.0.smithi191.stdout:1 : size label updated to 1073741824 2021-05-12T12:16:56.848 INFO:tasks.workunit.client.0.smithi191.stdout:2 : expanding from 0x80000000 to 0x100000000 2021-05-12T12:16:57.083 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:16:57.080+0000 7f60b4c52400 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-05-12T12:16:57.083 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:16:57.080+0000 7f60b4c52400 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-05-12T12:17:02.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2021-05-12T12:17:02.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2021-05-12T12:17:02.225 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:02.970 INFO:tasks.workunit.client.0.smithi191.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-05-12T12:17:02.971 INFO:tasks.workunit.client.0.smithi191.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T12:17:02.971 INFO:tasks.workunit.client.0.smithi191.stdout:2 : device size 0x101000000 : using 0x814cc000(2.0 GiB) 2021-05-12T12:17:02.971 INFO:tasks.workunit.client.0.smithi191.stdout:Expanding DB/WAL... 2021-05-12T12:17:02.971 INFO:tasks.workunit.client.0.smithi191.stdout:0 : expanding to 0x536870912 2021-05-12T12:17:02.997 INFO:tasks.workunit.client.0.smithi191.stdout:0 : size label updated to 536870912 2021-05-12T12:17:02.997 INFO:tasks.workunit.client.0.smithi191.stdout:1 : expanding to 0x1073741824 2021-05-12T12:17:03.014 INFO:tasks.workunit.client.0.smithi191.stdout:1 : size label updated to 1073741824 2021-05-12T12:17:03.014 INFO:tasks.workunit.client.0.smithi191.stdout:2 : expanding from 0x80000000 to 0x101000000 2021-05-12T12:17:03.241 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:03.238+0000 7f56836f5400 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-05-12T12:17:03.241 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:03.238+0000 7f56836f5400 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-05-12T12:17:08.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2021-05-12T12:17:08.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2021-05-12T12:17:08.355 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:09.142 INFO:tasks.workunit.client.0.smithi191.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-05-12T12:17:09.142 INFO:tasks.workunit.client.0.smithi191.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T12:17:09.142 INFO:tasks.workunit.client.0.smithi191.stdout:2 : device size 0x100020000 : using 0x80509000(2.0 GiB) 2021-05-12T12:17:09.143 INFO:tasks.workunit.client.0.smithi191.stdout:Expanding DB/WAL... 2021-05-12T12:17:09.143 INFO:tasks.workunit.client.0.smithi191.stdout:0 : expanding to 0x536870912 2021-05-12T12:17:09.179 INFO:tasks.workunit.client.0.smithi191.stdout:0 : size label updated to 536870912 2021-05-12T12:17:09.180 INFO:tasks.workunit.client.0.smithi191.stdout:1 : expanding to 0x1073741824 2021-05-12T12:17:09.205 INFO:tasks.workunit.client.0.smithi191.stdout:1 : size label updated to 1073741824 2021-05-12T12:17:09.205 INFO:tasks.workunit.client.0.smithi191.stdout:2 : expanding from 0x80000000 to 0x100020000 2021-05-12T12:17:09.456 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:09.453+0000 7f3b95e0c400 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-05-12T12:17:09.457 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:09.453+0000 7f3b95e0c400 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-05-12T12:17:14.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2021-05-12T12:17:14.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2021-05-12T12:17:14.427 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:15.214 INFO:tasks.workunit.client.0.smithi191.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-05-12T12:17:15.214 INFO:tasks.workunit.client.0.smithi191.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T12:17:15.215 INFO:tasks.workunit.client.0.smithi191.stdout:2 : device size 0xfff00000 : using 0x803ca000(2.0 GiB) 2021-05-12T12:17:15.215 INFO:tasks.workunit.client.0.smithi191.stdout:Expanding DB/WAL... 2021-05-12T12:17:15.215 INFO:tasks.workunit.client.0.smithi191.stdout:0 : expanding to 0x536870912 2021-05-12T12:17:15.245 INFO:tasks.workunit.client.0.smithi191.stdout:0 : size label updated to 536870912 2021-05-12T12:17:15.246 INFO:tasks.workunit.client.0.smithi191.stdout:1 : expanding to 0x1073741824 2021-05-12T12:17:15.270 INFO:tasks.workunit.client.0.smithi191.stdout:1 : size label updated to 1073741824 2021-05-12T12:17:15.270 INFO:tasks.workunit.client.0.smithi191.stdout:2 : expanding from 0x80000000 to 0xfff00000 2021-05-12T12:17:15.497 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:15.494+0000 7f35a7172400 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-05-12T12:17:15.497 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:15.494+0000 7f35a7172400 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-05-12T12:17:20.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:17:22.490 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:22.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:17:24.280 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:24.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:17:26.066 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:26.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:17:27.853 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:27.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2021-05-12T12:17:27.868 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:29.640 INFO:tasks.workunit.client.0.smithi191.stdout:0 : device size 0x1ffff000 : using 0x500000(5 MiB) 2021-05-12T12:17:29.641 INFO:tasks.workunit.client.0.smithi191.stdout:1 : device size 0x3fffe000 : using 0x1000000(16 MiB) 2021-05-12T12:17:29.641 INFO:tasks.workunit.client.0.smithi191.stdout:2 : device size 0x100000000 : using 0x4a8000(4.7 MiB) 2021-05-12T12:17:29.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2021-05-12T12:17:29.654 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:31.178 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-05-12T12:17:31.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:17:32.464 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:32.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-05-12T12:17:32.479 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:34.249 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-05-12T12:17:34.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:17:35.535 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:35.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2021-05-12T12:17:35.550 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:37.320 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-05-12T12:17:37.320 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-05-12T12:17:37.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:17:38.105 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:38.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2021-05-12T12:17:38.120 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:38.125 INFO:tasks.workunit.client.0.smithi191.stderr:Migrate to WAL device isn't supported. 2021-05-12T12:17:38.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2021-05-12T12:17:38.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:17:39.910 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:39.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-05-12T12:17:39.925 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:41.695 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-05-12T12:17:41.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:17:42.981 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:42.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2021-05-12T12:17:42.996 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:44.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:17:45.552 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:17:45.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-05-12T12:17:45.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:17:45.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:17:45.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:17:45.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:17:45.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T12:17:45.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:17:45.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:17:45.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:17:45.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:17:45.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T12:17:45.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T12:17:45.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:17:45.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:17:45.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:17:45.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:17:45.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:17:45.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:17:45.563 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:17:45.564 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:17:45.564 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:17:45.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:17:45.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:17:45.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:17:45.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:17:45.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:17:45.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:17:45.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:17:45.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:45.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:17:45.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:17:45.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T12:17:45.567 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:17:45.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:17:45.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:45.569 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-05-12T12:17:45.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:17:45.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:17:45.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:17:45.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:17:45.580 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:45.577+0000 7fe8035060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:45.583 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:45.580+0000 7fe8035060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:45.585 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:45.582+0000 7fe8035060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:45.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:17:45.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:17:45.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:17:45.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:17:45.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:17:45.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:45.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:17:45.928 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:17:45.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:45.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:46.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:46.351 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:46.349+0000 7fe8035060c0 -1 Falling back to public interface 2021-05-12T12:17:47.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:47.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:47.298 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:17:47.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:17:47.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:47.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:47.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:48.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:48.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:48.673 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:17:48.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:17:48.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:48.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:49.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:50.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:50.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:50.038 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:17:50.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:17:50.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:50.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:50.320 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:50.317+0000 7fe8035060c0 -1 osd.0 28 log_to_monitors {default=true} 2021-05-12T12:17:50.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:51.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:51.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:51.412 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:17:51.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:17:51.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:51.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:51.788 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/120781,v1:127.0.0.1:6803/120781] [v2:127.0.0.1:6804/120781,v1:127.0.0.1:6805/120781] exists,up d2d373bf-6a2c-4915-afa2-7895259f1e1d 2021-05-12T12:17:51.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:17:51.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:17:51.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:17:51.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T12:17:51.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=120781 2021-05-12T12:17:51.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-05-12T12:17:51.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:17:51.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:17:51.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:17:51.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:17:51.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T12:17:51.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:17:51.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:17:51.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:17:51.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:17:51.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T12:17:51.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T12:17:51.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:17:51.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:17:51.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:17:51.797 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:17:51.797 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:17:51.798 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:17:51.798 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:17:51.798 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:17:51.798 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:17:51.798 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:17:51.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:17:51.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:17:51.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:17:51.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:17:51.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:17:51.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:17:51.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:51.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:17:51.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:17:51.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T12:17:51.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:17:51.801 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:17:51.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:51.802 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-05-12T12:17:51.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:17:51.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:17:51.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:17:51.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:17:51.812 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:51.809+0000 7fb8667ea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:51.814 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:51.811+0000 7fb8667ea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:51.815 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:51.812+0000 7fb8667ea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:52.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:17:52.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:17:52.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:17:52.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:17:52.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:17:52.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:52.154 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:17:52.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:17:52.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:52.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:17:52.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:52.585 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:52.582+0000 7fb8667ea0c0 -1 Falling back to public interface 2021-05-12T12:17:53.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:53.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:53.524 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:17:53.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:17:53.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:53.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:17:53.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:54.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:54.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:54.887 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:17:54.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:17:54.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:54.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:17:55.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:56.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:56.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:56.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:17:56.246 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:17:56.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:56.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:17:56.540 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:56.537+0000 7fb8667ea0c0 -1 osd.1 31 log_to_monitors {default=true} 2021-05-12T12:17:56.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:57.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:57.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:57.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:17:57.621 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:17:57.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:57.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:17:58.065 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/121768,v1:127.0.0.1:6811/121768] [v2:127.0.0.1:6812/121768,v1:127.0.0.1:6813/121768] exists,up f71f7391-473d-4cc3-8e29-43a620de60a1 2021-05-12T12:17:58.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:17:58.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:17:58.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:17:58.067 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T12:17:58.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=121768 2021-05-12T12:17:58.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-05-12T12:17:58.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:17:58.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:17:58.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:17:58.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:17:58.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T12:17:58.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:17:58.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:17:58.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:17:58.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:17:58.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T12:17:58.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T12:17:58.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:17:58.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:17:58.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:17:58.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:17:58.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:17:58.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:17:58.075 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:17:58.075 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:17:58.075 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:17:58.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:17:58.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:17:58.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:17:58.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:17:58.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:17:58.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:17:58.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:17:58.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:58.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:17:58.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:17:58.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T12:17:58.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:17:58.078 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:17:58.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:58.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-05-12T12:17:58.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:17:58.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:17:58.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:17:58.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:17:58.089 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:58.086+0000 7f42898fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:58.091 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:58.088+0000 7f42898fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:58.093 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:58.090+0000 7f42898fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:58.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:17:58.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:17:58.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:17:58.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:17:58.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:17:58.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:58.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:17:58.445 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:17:58.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:58.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:17:58.609 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:17:58.606+0000 7f42898fd0c0 -1 Falling back to public interface 2021-05-12T12:17:58.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:59.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:59.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:59.814 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:17:59.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:17:59.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:59.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:18:00.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:01.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:01.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:01.171 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:18:01.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:18:01.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:01.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:18:01.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:02.032 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:18:02.029+0000 7f42898fd0c0 -1 osd.2 34 log_to_monitors {default=true} 2021-05-12T12:18:02.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:02.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:02.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:18:02.563 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:18:02.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:02.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:18:03.028 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 46 up_thru 32 down_at 35 last_clean_interval [15,34) [v2:127.0.0.1:6818/122755,v1:127.0.0.1:6819/122755] [v2:127.0.0.1:6820/122755,v1:127.0.0.1:6821/122755] exists,up ad18b5ee-7ebd-47ea-8856-c1d530f45233 2021-05-12T12:18:03.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:18:03.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:18:03.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:18:03.029 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T12:18:03.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=122755 2021-05-12T12:18:03.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-05-12T12:18:03.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:18:03.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:18:03.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T12:18:03.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:18:03.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T12:18:03.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:18:03.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:18:03.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:18:03.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:18:03.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T12:18:03.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T12:18:03.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:18:03.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:18:03.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:18:03.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:18:03.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:18:03.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:18:03.037 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:18:03.037 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:18:03.037 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:18:03.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:18:03.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:18:03.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:18:03.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:18:03.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:18:03.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:18:03.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:18:03.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:18:03.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:18:03.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:18:03.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T12:18:03.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T12:18:03.040 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.3 2021-05-12T12:18:03.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:18:03.042 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-05-12T12:18:03.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T12:18:03.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:18:03.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:18:03.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:18:03.052 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:18:03.049+0000 7f84083f20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:18:03.054 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:18:03.051+0000 7f84083f20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:18:03.056 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:18:03.052+0000 7f84083f20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:18:03.746 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:18:03.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T12:18:03.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:18:03.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:18:03.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:18:03.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:18:03.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:03.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:18:03.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:03.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:18:03.824 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:18:03.820+0000 7f84083f20c0 -1 Falling back to public interface 2021-05-12T12:18:04.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:05.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:05.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:05.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:18:05.207 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:18:05.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:05.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:18:05.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:06.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:06.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:06.579 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:18:06.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:18:06.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:06.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:18:06.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:07.440 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:18:07.437+0000 7f84083f20c0 -1 osd.3 36 log_to_monitors {default=true} 2021-05-12T12:18:07.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:07.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:07.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:18:07.939 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:18:07.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:07.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:18:08.320 INFO:tasks.workunit.client.0.smithi191.stdout:osd.3 up in weight 1 up_from 50 up_thru 35 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/123706,v1:127.0.0.1:6827/123706] [v2:127.0.0.1:6828/123706,v1:127.0.0.1:6829/123706] exists,up 16258e95-8742-4373-96ca-e9d8bda153b0 2021-05-12T12:18:08.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:18:08.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:18:08.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:18:08.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T12:18:08.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=123706 2021-05-12T12:18:08.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2021-05-12T12:18:08.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:18:08.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:18:08.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:18:08.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:18:08.325 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:18:08.325 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:18:08.325 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:18:08.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:18:08.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:18:08.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:18:08.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:18:08.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:18:08.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:18:08.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:18:08.410 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:18:08.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:18:08.851 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:18:08.851 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:18:08.851 INFO:tasks.workunit.client.0.smithi191.stderr:3' 2021-05-12T12:18:08.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:18:08.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:18:08.852 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:18:08.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=167503724550 2021-05-12T12:18:08.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 167503724550 2021-05-12T12:18:08.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-167503724550' 2021-05-12T12:18:08.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:18:08.983 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:18:09.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=180388626437 2021-05-12T12:18:09.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 180388626437 2021-05-12T12:18:09.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-167503724550 1-180388626437' 2021-05-12T12:18:09.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:18:09.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:18:09.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=197568495620 2021-05-12T12:18:09.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 197568495620 2021-05-12T12:18:09.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-167503724550 1-180388626437 2-197568495620' 2021-05-12T12:18:09.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:18:09.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:18:09.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=214748364803 2021-05-12T12:18:09.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 214748364803 2021-05-12T12:18:09.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-167503724550 1-180388626437 2-197568495620 3-214748364803' 2021-05-12T12:18:09.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:18:09.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-167503724550 2021-05-12T12:18:09.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:18:09.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:18:09.390 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-167503724550 2021-05-12T12:18:09.390 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:18:09.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=167503724550 2021-05-12T12:18:09.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 167503724550' 2021-05-12T12:18:09.391 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 167503724550 2021-05-12T12:18:09.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:18:09.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 167503724548 -lt 167503724550 2021-05-12T12:18:09.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:18:10.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:18:10.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:18:11.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 167503724550 -lt 167503724550 2021-05-12T12:18:11.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:18:11.121 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-180388626437 2021-05-12T12:18:11.121 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:18:11.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:18:11.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-180388626437 2021-05-12T12:18:11.123 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:18:11.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=180388626437 2021-05-12T12:18:11.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 180388626437' 2021-05-12T12:18:11.124 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 180388626437 2021-05-12T12:18:11.124 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:18:11.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 180388626437 -lt 180388626437 2021-05-12T12:18:11.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:18:11.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-197568495620 2021-05-12T12:18:11.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:18:11.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:18:11.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-197568495620 2021-05-12T12:18:11.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:18:11.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=197568495620 2021-05-12T12:18:11.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 197568495620' 2021-05-12T12:18:11.489 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 197568495620 2021-05-12T12:18:11.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:18:11.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 197568495620 -lt 197568495620 2021-05-12T12:18:11.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:18:11.852 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-214748364803 2021-05-12T12:18:11.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:18:11.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:18:11.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-214748364803 2021-05-12T12:18:11.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:18:11.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=214748364803 2021-05-12T12:18:11.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 214748364803' 2021-05-12T12:18:11.856 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 214748364803 2021-05-12T12:18:11.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:18:12.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 214748364803 -lt 214748364803 2021-05-12T12:18:12.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:18:12.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:18:12.214 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:18:12.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 16 == 0 2021-05-12T12:18:12.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:18:12.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:18:12.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:18:12.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:18:12.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:18:12.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:18:12.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:18:13.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=16 2021-05-12T12:18:13.045 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:18:13.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:18:13.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:18:13.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 16 = 16 2021-05-12T12:18:13.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:18:13.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:18:13.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T12:18:13.544 INFO:tasks.workunit.client.0.smithi191.stdout:hints = 1 2021-05-12T12:18:13.544 INFO:tasks.workunit.client.0.smithi191.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T12:18:13.544 INFO:tasks.workunit.client.0.smithi191.stdout:Object prefix: benchmark_data_smithi191_125099 2021-05-12T12:18:13.545 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:18:13.545 INFO:tasks.workunit.client.0.smithi191.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:18:14.544 INFO:tasks.workunit.client.0.smithi191.stdout: 1 16 38 22 0.085929 0.0859375 0.308374 0.324566 2021-05-12T12:18:15.544 INFO:tasks.workunit.client.0.smithi191.stdout: 2 16 47 31 0.060541 0.0351562 0.0841154 0.279281 2021-05-12T12:18:16.544 INFO:tasks.workunit.client.0.smithi191.stdout: 3 16 51 35 0.0455678 0.015625 0.142192 0.260045 2021-05-12T12:18:17.544 INFO:tasks.workunit.client.0.smithi191.stdout: 4 16 51 35 0.0341756 0 - 0.260045 2021-05-12T12:18:18.544 INFO:tasks.workunit.client.0.smithi191.stdout: 5 16 51 35 0.0273407 0 - 0.260045 2021-05-12T12:18:19.544 INFO:tasks.workunit.client.0.smithi191.stdout: 6 16 51 35 0.0227839 0 - 0.260045 2021-05-12T12:18:20.545 INFO:tasks.workunit.client.0.smithi191.stdout: 7 16 69 53 0.0295725 0.0175781 0.302016 1.25605 2021-05-12T12:18:21.545 INFO:tasks.workunit.client.0.smithi191.stdout: 8 16 91 75 0.0366168 0.0859375 0.301705 0.980892 2021-05-12T12:18:22.545 INFO:tasks.workunit.client.0.smithi191.stdout: 9 16 116 100 0.0433976 0.0976562 0.340116 1.41756 2021-05-12T12:18:23.545 INFO:tasks.workunit.client.0.smithi191.stdout: 10 16 163 147 0.0574152 0.183594 0.33559 1.07156 2021-05-12T12:18:24.545 INFO:tasks.workunit.client.0.smithi191.stdout: 11 16 211 195 0.0692391 0.1875 0.326369 0.888637 2021-05-12T12:18:25.545 INFO:tasks.workunit.client.0.smithi191.stdout: 12 16 264 248 0.08072 0.207031 0.243178 0.764263 2021-05-12T12:18:26.545 INFO:tasks.workunit.client.0.smithi191.stdout: 13 16 320 304 0.0913355 0.21875 0.308817 0.67769 2021-05-12T12:18:27.545 INFO:tasks.workunit.client.0.smithi191.stdout: 14 16 365 349 0.0973657 0.175781 0.298116 0.634578 2021-05-12T12:18:28.546 INFO:tasks.workunit.client.0.smithi191.stdout: 15 16 417 401 0.104415 0.203125 0.319185 0.590686 2021-05-12T12:18:29.546 INFO:tasks.workunit.client.0.smithi191.stdout: 16 16 473 457 0.111559 0.21875 0.277349 0.554983 2021-05-12T12:18:30.546 INFO:tasks.workunit.client.0.smithi191.stdout: 17 16 526 510 0.117174 0.207031 0.281471 0.528503 2021-05-12T12:18:31.546 INFO:tasks.workunit.client.0.smithi191.stdout: 18 16 578 562 0.121948 0.203125 0.276022 0.507703 2021-05-12T12:18:32.546 INFO:tasks.workunit.client.0.smithi191.stdout: 19 16 617 601 0.123547 0.152344 0.269095 0.501344 2021-05-12T12:18:33.546 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T12:18:33.544349+0000 min lat: 0.0754343 max lat: 8.83057 avg lat: 0.48372 2021-05-12T12:18:33.546 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:18:33.547 INFO:tasks.workunit.client.0.smithi191.stdout: 20 16 674 658 0.128501 0.222656 0.335696 0.48372 2021-05-12T12:18:34.546 INFO:tasks.workunit.client.0.smithi191.stdout: 21 16 724 708 0.131681 0.195312 0.30266 0.472222 2021-05-12T12:18:35.546 INFO:tasks.workunit.client.0.smithi191.stdout: 22 16 773 757 0.134395 0.191406 0.349427 0.462361 2021-05-12T12:18:36.546 INFO:tasks.workunit.client.0.smithi191.stdout: 23 16 822 806 0.136873 0.191406 0.293729 0.453232 2021-05-12T12:18:37.547 INFO:tasks.workunit.client.0.smithi191.stdout: 24 16 868 852 0.138656 0.179688 0.448283 0.445871 2021-05-12T12:18:38.547 INFO:tasks.workunit.client.0.smithi191.stdout: 25 16 923 907 0.141702 0.214844 0.251877 0.437905 2021-05-12T12:18:39.547 INFO:tasks.workunit.client.0.smithi191.stdout: 26 16 972 956 0.143613 0.191406 0.263885 0.431981 2021-05-12T12:18:40.547 INFO:tasks.workunit.client.0.smithi191.stdout: 27 16 1018 1002 0.144948 0.179688 0.33751 0.427625 2021-05-12T12:18:41.547 INFO:tasks.workunit.client.0.smithi191.stdout: 28 16 1069 1053 0.146885 0.199219 0.293736 0.424007 2021-05-12T12:18:42.547 INFO:tasks.workunit.client.0.smithi191.stdout: 29 16 1116 1100 0.14815 0.183594 0.310206 0.419135 2021-05-12T12:18:43.547 INFO:tasks.workunit.client.0.smithi191.stdout: 30 15 1170 1155 0.150372 0.214844 0.258382 0.414291 2021-05-12T12:18:44.548 INFO:tasks.workunit.client.0.smithi191.stdout:Total time run: 30.2353 2021-05-12T12:18:44.548 INFO:tasks.workunit.client.0.smithi191.stdout:Total writes made: 1170 2021-05-12T12:18:44.548 INFO:tasks.workunit.client.0.smithi191.stdout:Write size: 4096 2021-05-12T12:18:44.549 INFO:tasks.workunit.client.0.smithi191.stdout:Object size: 4096 2021-05-12T12:18:44.549 INFO:tasks.workunit.client.0.smithi191.stdout:Bandwidth (MB/sec): 0.151158 2021-05-12T12:18:44.549 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Bandwidth: 0.0783845 2021-05-12T12:18:44.549 INFO:tasks.workunit.client.0.smithi191.stdout:Max bandwidth (MB/sec): 0.222656 2021-05-12T12:18:44.549 INFO:tasks.workunit.client.0.smithi191.stdout:Min bandwidth (MB/sec): 0 2021-05-12T12:18:44.550 INFO:tasks.workunit.client.0.smithi191.stdout:Average IOPS: 38 2021-05-12T12:18:44.550 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev IOPS: 20.0954 2021-05-12T12:18:44.550 INFO:tasks.workunit.client.0.smithi191.stdout:Max IOPS: 57 2021-05-12T12:18:44.550 INFO:tasks.workunit.client.0.smithi191.stdout:Min IOPS: 0 2021-05-12T12:18:44.550 INFO:tasks.workunit.client.0.smithi191.stdout:Average Latency(s): 0.41266 2021-05-12T12:18:44.551 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Latency(s): 0.781637 2021-05-12T12:18:44.551 INFO:tasks.workunit.client.0.smithi191.stdout:Max latency(s): 8.83057 2021-05-12T12:18:44.551 INFO:tasks.workunit.client.0.smithi191.stdout:Min latency(s): 0.0754343 2021-05-12T12:18:44.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 120781 2021-05-12T12:18:44.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2021-05-12T12:18:45.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 120781 2021-05-12T12:18:45.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (120781) - No such process 2021-05-12T12:18:45.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2021-05-12T12:18:46.068 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already down. 2021-05-12T12:18:46.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 121768 2021-05-12T12:18:46.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2021-05-12T12:18:47.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 121768 2021-05-12T12:18:47.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (121768) - No such process 2021-05-12T12:18:47.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2021-05-12T12:18:47.605 INFO:tasks.workunit.client.0.smithi191.stderr:osd.1 is already down. 2021-05-12T12:18:47.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 122755 2021-05-12T12:18:47.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2021-05-12T12:18:48.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 122755 2021-05-12T12:18:48.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (122755) - No such process 2021-05-12T12:18:48.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2021-05-12T12:18:49.033 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already down. 2021-05-12T12:18:49.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 123706 2021-05-12T12:18:49.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2021-05-12T12:18:50.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 123706 2021-05-12T12:18:50.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (123706) - No such process 2021-05-12T12:18:50.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2021-05-12T12:18:50.548 INFO:tasks.workunit.client.0.smithi191.stderr:osd.3 is already down. 2021-05-12T12:18:50.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:18:51.838 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:18:51.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2021-05-12T12:18:52.090 INFO:tasks.workunit.client.0.smithi191.stderr:512+0 records in 2021-05-12T12:18:52.091 INFO:tasks.workunit.client.0.smithi191.stderr:512+0 records out 2021-05-12T12:18:52.091 INFO:tasks.workunit.client.0.smithi191.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.248317 s, 2.2 GB/s 2021-05-12T12:18:52.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2021-05-12T12:18:52.101 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:18:57.121 INFO:tasks.workunit.client.0.smithi191.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2021-05-12T12:18:57.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:18:58.906 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:18:58.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:19:00.192 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:19:00.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2021-05-12T12:19:00.677 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records in 2021-05-12T12:19:00.678 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records out 2021-05-12T12:19:00.678 INFO:tasks.workunit.client.0.smithi191.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.481613 s, 2.2 GB/s 2021-05-12T12:19:00.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2021-05-12T12:19:00.687 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:09.710 INFO:tasks.workunit.client.0.smithi191.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2021-05-12T12:19:09.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2021-05-12T12:19:09.724 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:11.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:19:13.282 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:19:13.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:19:14.068 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:19:14.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2021-05-12T12:19:14.083 INFO:tasks.workunit.client.0.smithi191.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-05-12T12:19:14.083 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:14.852 INFO:tasks.workunit.client.0.smithi191.stdout:DB device added 2021-05-12T12:19:14.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2021-05-12T12:19:14.867 INFO:tasks.workunit.client.0.smithi191.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-05-12T12:19:14.868 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:16.137 INFO:tasks.workunit.client.0.smithi191.stdout:WAL device added 2021-05-12T12:19:16.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2021-05-12T12:19:16.152 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:17.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:19:19.710 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:19:19.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:19:20.997 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:19:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2021-05-12T12:19:21.012 INFO:tasks.workunit.client.0.smithi191.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-05-12T12:19:21.012 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:22.282 INFO:tasks.workunit.client.0.smithi191.stdout:WAL device added 2021-05-12T12:19:22.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-05-12T12:19:22.297 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:24.067 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2021-05-12T12:19:24.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:19:25.354 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:19:25.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-05-12T12:19:25.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:19:25.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:19:25.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:19:25.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:19:25.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T12:19:25.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:19:25.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:19:25.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:19:25.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:19:25.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T12:19:25.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T12:19:25.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:19:25.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:19:25.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:19:25.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:19:25.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:19:25.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:19:25.364 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:19:25.364 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:19:25.364 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:19:25.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:19:25.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:19:25.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:19:25.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:19:25.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:19:25.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:19:25.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:19:25.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:25.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:19:25.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:19:25.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T12:19:25.367 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:19:25.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:19:25.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:25.369 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-05-12T12:19:25.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:19:25.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:19:25.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:19:25.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:19:25.383 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:25.380+0000 7fa8bef930c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:25.385 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:25.382+0000 7fa8bef930c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:25.386 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:25.383+0000 7fa8bef930c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:25.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:19:25.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:19:25.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:19:25.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:19:25.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:19:25.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:25.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:19:25.738 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:19:25.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:25.739 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:26.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:26.407 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:26.404+0000 7fa8bef930c0 -1 Falling back to public interface 2021-05-12T12:19:27.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:27.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:27.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:19:27.094 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:19:27.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:27.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:27.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:28.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:28.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:28.458 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:19:28.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:19:28.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:28.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:28.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:29.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:29.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:29.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:19:29.827 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:19:29.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:29.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:30.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:31.093 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:31.089+0000 7fa8bef930c0 -1 osd.0 52 log_to_monitors {default=true} 2021-05-12T12:19:31.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:31.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:31.190 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:19:31.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:19:31.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:31.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:31.553 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 62 up_thru 50 down_at 53 last_clean_interval [39,52) [v2:127.0.0.1:6802/131305,v1:127.0.0.1:6803/131305] [v2:127.0.0.1:6804/131305,v1:127.0.0.1:6805/131305] exists,up d2d373bf-6a2c-4915-afa2-7895259f1e1d 2021-05-12T12:19:31.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:19:31.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:19:31.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:19:31.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T12:19:31.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=131305 2021-05-12T12:19:31.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-05-12T12:19:31.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:19:31.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:19:31.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:19:31.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:19:31.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T12:19:31.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:19:31.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:19:31.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:19:31.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:19:31.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T12:19:31.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T12:19:31.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:19:31.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:19:31.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:19:31.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:19:31.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:19:31.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:19:31.566 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:19:31.566 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:19:31.566 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:19:31.567 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:19:31.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:19:31.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:19:31.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:19:31.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:19:31.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:19:31.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:19:31.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:31.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:19:31.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:19:31.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T12:19:31.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:19:31.569 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:19:31.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:31.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-05-12T12:19:31.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:19:31.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:19:31.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:19:31.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:19:31.577 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:31.574+0000 7efc031b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:31.579 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:31.576+0000 7efc031b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:31.581 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:31.578+0000 7efc031b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:31.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:19:31.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:19:31.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:19:31.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:19:31.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:19:31.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:31.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:19:31.941 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:19:31.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:31.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:32.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:32.597 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:32.594+0000 7efc031b90c0 -1 Falling back to public interface 2021-05-12T12:19:33.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:33.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:33.299 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:19:33.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:19:33.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:33.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:33.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:34.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:34.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:34.665 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:19:34.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:19:34.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:34.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:35.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:36.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:36.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:36.076 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:19:36.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:19:36.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:36.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:36.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:37.122 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:37.119+0000 7efc031b90c0 -1 osd.1 54 log_to_monitors {default=true} 2021-05-12T12:19:37.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:37.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:37.441 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:19:37.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:19:37.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:37.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:37.932 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 55 last_clean_interval [42,54) [v2:127.0.0.1:6810/132302,v1:127.0.0.1:6811/132302] [v2:127.0.0.1:6812/132302,v1:127.0.0.1:6813/132302] exists,up f71f7391-473d-4cc3-8e29-43a620de60a1 2021-05-12T12:19:37.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:19:37.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:19:37.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:19:37.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T12:19:37.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=132302 2021-05-12T12:19:37.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-05-12T12:19:37.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:19:37.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:19:37.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:19:37.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:19:37.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T12:19:37.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:19:37.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:19:37.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:19:37.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:19:37.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T12:19:37.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T12:19:37.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:19:37.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:19:37.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:19:37.942 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:19:37.942 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:19:37.942 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:19:37.942 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:19:37.942 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:19:37.943 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:19:37.943 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:19:37.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:19:37.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:19:37.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:19:37.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:19:37.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:19:37.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:19:37.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:37.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:19:37.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:19:37.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T12:19:37.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:19:37.946 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:19:37.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:37.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-05-12T12:19:37.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:19:37.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:19:37.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:19:37.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:19:37.956 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:37.953+0000 7f852576e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:37.975 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:37.972+0000 7f852576e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:37.977 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:37.974+0000 7f852576e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:38.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:19:38.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:19:38.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:19:38.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:19:38.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:19:38.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:38.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:19:38.307 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:19:38.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:38.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:38.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:38.993 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:38.990+0000 7f852576e0c0 -1 Falling back to public interface 2021-05-12T12:19:39.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:39.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:39.667 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:19:39.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:19:39.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:39.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:40.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:41.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:41.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:41.031 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:19:41.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:19:41.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:41.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:41.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:42.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:42.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:42.399 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:19:42.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:19:42.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:42.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:42.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:43.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:43.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:43.769 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:19:43.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:19:43.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:43.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:44.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:44.159 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:44.156+0000 7f852576e0c0 -1 osd.2 57 log_to_monitors {default=true} 2021-05-12T12:19:45.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:45.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:45.151 INFO:tasks.workunit.client.0.smithi191.stdout:5 2021-05-12T12:19:45.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 5 2021-05-12T12:19:45.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:45.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:45.987 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 68 up_thru 68 down_at 58 last_clean_interval [46,57) [v2:127.0.0.1:6818/133299,v1:127.0.0.1:6819/133299] [v2:127.0.0.1:6820/133299,v1:127.0.0.1:6821/133299] exists,up ad18b5ee-7ebd-47ea-8856-c1d530f45233 2021-05-12T12:19:45.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:19:45.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:19:45.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:19:45.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T12:19:45.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=133299 2021-05-12T12:19:45.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-05-12T12:19:45.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:19:45.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:19:45.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T12:19:45.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:19:45.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T12:19:45.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:19:45.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:19:45.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:19:45.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:19:45.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T12:19:45.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T12:19:45.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:19:45.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:19:45.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:19:45.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:19:45.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:19:45.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:19:45.997 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:19:45.997 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:19:45.998 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:19:45.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:19:45.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:19:45.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:19:45.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:19:45.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:19:45.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:19:45.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:19:45.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:46.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:19:46.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:19:46.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T12:19:46.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T12:19:46.001 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.3 2021-05-12T12:19:46.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:46.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-05-12T12:19:46.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T12:19:46.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:19:46.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:19:46.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:19:46.027 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:46.023+0000 7f5d0bead0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:46.049 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:46.046+0000 7f5d0bead0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:46.053 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:46.048+0000 7f5d0bead0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:46.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T12:19:46.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:19:46.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:19:46.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:19:46.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:19:46.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:46.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:19:46.410 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:19:46.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:46.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:46.573 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:46.570+0000 7f5d0bead0c0 -1 Falling back to public interface 2021-05-12T12:19:46.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:47.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:47.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:47.809 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:19:47.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:19:47.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:47.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:48.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:49.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:49.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:49.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:19:49.181 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:19:49.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:49.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:49.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:50.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:50.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:50.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:19:50.575 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:19:50.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:50.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:50.895 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:19:50.892+0000 7f5d0bead0c0 -1 osd.3 59 log_to_monitors {default=true} 2021-05-12T12:19:50.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:51.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:51.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:51.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:19:51.973 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:19:51.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:51.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:52.602 INFO:tasks.workunit.client.0.smithi191.stdout:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/134322,v1:127.0.0.1:6827/134322] [v2:127.0.0.1:6828/134322,v1:127.0.0.1:6829/134322] exists,up 16258e95-8742-4373-96ca-e9d8bda153b0 2021-05-12T12:19:52.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:19:52.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:19:52.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:19:52.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T12:19:52.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=134322 2021-05-12T12:19:52.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T12:19:52.769 INFO:tasks.workunit.client.0.smithi191.stdout:hints = 1 2021-05-12T12:19:52.769 INFO:tasks.workunit.client.0.smithi191.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T12:19:52.769 INFO:tasks.workunit.client.0.smithi191.stdout:Object prefix: benchmark_data_smithi191_135294 2021-05-12T12:19:52.769 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:19:52.770 INFO:tasks.workunit.client.0.smithi191.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:19:53.769 INFO:tasks.workunit.client.0.smithi191.stdout: 1 16 30 14 0.0546824 0.0546875 0.342917 0.565573 2021-05-12T12:19:54.769 INFO:tasks.workunit.client.0.smithi191.stdout: 2 16 41 25 0.0488235 0.0429688 0.259614 0.44198 2021-05-12T12:19:55.769 INFO:tasks.workunit.client.0.smithi191.stdout: 3 16 43 27 0.0351524 0.0078125 0.142311 0.423296 2021-05-12T12:19:56.769 INFO:tasks.workunit.client.0.smithi191.stdout: 4 16 43 27 0.0263641 0 - 0.423296 2021-05-12T12:19:57.769 INFO:tasks.workunit.client.0.smithi191.stdout: 5 16 43 27 0.0210911 0 - 0.423296 2021-05-12T12:19:58.769 INFO:tasks.workunit.client.0.smithi191.stdout: 6 16 43 27 0.0175759 0 - 0.423296 2021-05-12T12:19:59.772 INFO:tasks.workunit.client.0.smithi191.stdout: 7 16 43 27 0.0150651 0 - 0.423296 2021-05-12T12:20:00.769 INFO:tasks.workunit.client.0.smithi191.stdout: 8 16 43 27 0.013182 0 - 0.423296 2021-05-12T12:20:01.770 INFO:tasks.workunit.client.0.smithi191.stdout: 9 16 53 37 0.0160571 0.00651042 8.19008 2.54644 2021-05-12T12:20:02.770 INFO:tasks.workunit.client.0.smithi191.stdout: 10 16 88 72 0.0281216 0.136719 0.363211 2.16683 2021-05-12T12:20:03.770 INFO:tasks.workunit.client.0.smithi191.stdout: 11 16 133 117 0.0415434 0.175781 0.392877 1.47711 2021-05-12T12:20:04.770 INFO:tasks.workunit.client.0.smithi191.stdout: 12 16 183 167 0.0543554 0.195312 0.380953 1.13878 2021-05-12T12:20:05.770 INFO:tasks.workunit.client.0.smithi191.stdout: 13 16 213 197 0.0591875 0.117188 0.643149 1.02147 2021-05-12T12:20:06.770 INFO:tasks.workunit.client.0.smithi191.stdout: 14 16 261 245 0.0683509 0.1875 0.336203 0.898554 2021-05-12T12:20:07.770 INFO:tasks.workunit.client.0.smithi191.stdout: 15 16 308 292 0.0760321 0.183594 0.427866 0.812407 2021-05-12T12:20:08.770 INFO:tasks.workunit.client.0.smithi191.stdout: 16 16 346 330 0.0805563 0.148438 0.417688 0.761112 2021-05-12T12:20:09.770 INFO:tasks.workunit.client.0.smithi191.stdout: 17 16 394 378 0.0868459 0.1875 0.38371 0.712374 2021-05-12T12:20:10.771 INFO:tasks.workunit.client.0.smithi191.stdout: 18 16 442 426 0.0924365 0.1875 0.219662 0.67122 2021-05-12T12:20:11.771 INFO:tasks.workunit.client.0.smithi191.stdout: 19 16 477 461 0.0947661 0.136719 0.18516 0.650107 2021-05-12T12:20:12.771 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T12:20:12.769269+0000 min lat: 0.142311 max lat: 9.3534 avg lat: 0.624371 2021-05-12T12:20:12.771 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:20:12.772 INFO:tasks.workunit.client.0.smithi191.stdout: 20 16 521 505 0.0986204 0.171875 0.459812 0.624371 2021-05-12T12:20:13.771 INFO:tasks.workunit.client.0.smithi191.stdout: 21 16 568 552 0.102666 0.183594 0.404429 0.604159 2021-05-12T12:20:14.771 INFO:tasks.workunit.client.0.smithi191.stdout: 22 16 609 593 0.105278 0.160156 0.367985 0.587788 2021-05-12T12:20:15.771 INFO:tasks.workunit.client.0.smithi191.stdout: 23 16 648 632 0.107323 0.152344 0.382005 0.574077 2021-05-12T12:20:16.772 INFO:tasks.workunit.client.0.smithi191.stdout: 24 16 689 673 0.109524 0.160156 0.224608 0.565869 2021-05-12T12:20:17.772 INFO:tasks.workunit.client.0.smithi191.stdout: 25 16 733 717 0.112017 0.171875 0.348896 0.552658 2021-05-12T12:20:18.772 INFO:tasks.workunit.client.0.smithi191.stdout: 26 16 779 763 0.114619 0.179688 0.270935 0.541401 2021-05-12T12:20:19.772 INFO:tasks.workunit.client.0.smithi191.stdout: 27 16 820 804 0.116304 0.160156 0.259125 0.532721 2021-05-12T12:20:20.772 INFO:tasks.workunit.client.0.smithi191.stdout: 28 16 864 848 0.118288 0.171875 0.285014 0.524452 2021-05-12T12:20:21.772 INFO:tasks.workunit.client.0.smithi191.stdout: 29 16 908 892 0.120136 0.171875 0.401814 0.518147 2021-05-12T12:20:22.772 INFO:tasks.workunit.client.0.smithi191.stdout: 30 16 946 930 0.121078 0.148438 0.417908 0.512959 2021-05-12T12:20:23.773 INFO:tasks.workunit.client.0.smithi191.stdout:Total time run: 30.2979 2021-05-12T12:20:23.773 INFO:tasks.workunit.client.0.smithi191.stdout:Total writes made: 946 2021-05-12T12:20:23.773 INFO:tasks.workunit.client.0.smithi191.stdout:Write size: 4096 2021-05-12T12:20:23.773 INFO:tasks.workunit.client.0.smithi191.stdout:Object size: 4096 2021-05-12T12:20:23.774 INFO:tasks.workunit.client.0.smithi191.stdout:Bandwidth (MB/sec): 0.121966 2021-05-12T12:20:23.774 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Bandwidth: 0.0744174 2021-05-12T12:20:23.774 INFO:tasks.workunit.client.0.smithi191.stdout:Max bandwidth (MB/sec): 0.195312 2021-05-12T12:20:23.774 INFO:tasks.workunit.client.0.smithi191.stdout:Min bandwidth (MB/sec): 0 2021-05-12T12:20:23.774 INFO:tasks.workunit.client.0.smithi191.stdout:Average IOPS: 31 2021-05-12T12:20:23.775 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev IOPS: 19.0863 2021-05-12T12:20:23.775 INFO:tasks.workunit.client.0.smithi191.stdout:Max IOPS: 50 2021-05-12T12:20:23.775 INFO:tasks.workunit.client.0.smithi191.stdout:Min IOPS: 0 2021-05-12T12:20:23.775 INFO:tasks.workunit.client.0.smithi191.stdout:Average Latency(s): 0.511021 2021-05-12T12:20:23.775 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Latency(s): 1.04264 2021-05-12T12:20:23.776 INFO:tasks.workunit.client.0.smithi191.stdout:Max latency(s): 9.3534 2021-05-12T12:20:23.776 INFO:tasks.workunit.client.0.smithi191.stdout:Min latency(s): 0.142311 2021-05-12T12:20:23.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 131305 2021-05-12T12:20:23.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2021-05-12T12:20:24.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 131305 2021-05-12T12:20:24.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (131305) - No such process 2021-05-12T12:20:24.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2021-05-12T12:20:25.369 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already down. 2021-05-12T12:20:25.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 132302 2021-05-12T12:20:25.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2021-05-12T12:20:26.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 132302 2021-05-12T12:20:26.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (132302) - No such process 2021-05-12T12:20:26.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2021-05-12T12:20:26.811 INFO:tasks.workunit.client.0.smithi191.stderr:osd.1 is already down. 2021-05-12T12:20:26.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 133299 2021-05-12T12:20:26.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2021-05-12T12:20:27.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 133299 2021-05-12T12:20:27.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (133299) - No such process 2021-05-12T12:20:27.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2021-05-12T12:20:28.209 INFO:tasks.workunit.client.0.smithi191.stderr:osd.2 is already down. 2021-05-12T12:20:28.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 134322 2021-05-12T12:20:28.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2021-05-12T12:20:29.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 134322 2021-05-12T12:20:29.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (134322) - No such process 2021-05-12T12:20:29.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2021-05-12T12:20:29.771 INFO:tasks.workunit.client.0.smithi191.stderr:osd.3 is already down. 2021-05-12T12:20:29.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:20:31.564 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:20:31.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2021-05-12T12:20:32.049 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records in 2021-05-12T12:20:32.049 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records out 2021-05-12T12:20:32.049 INFO:tasks.workunit.client.0.smithi191.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.480865 s, 2.2 GB/s 2021-05-12T12:20:32.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2021-05-12T12:20:32.059 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:20:42.081 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2021-05-12T12:20:42.081 INFO:tasks.workunit.client.0.smithi191.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2021-05-12T12:20:42.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:20:43.868 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:20:43.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2021-05-12T12:20:44.126 INFO:tasks.workunit.client.0.smithi191.stderr:512+0 records in 2021-05-12T12:20:44.126 INFO:tasks.workunit.client.0.smithi191.stderr:512+0 records out 2021-05-12T12:20:44.126 INFO:tasks.workunit.client.0.smithi191.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.254174 s, 2.1 GB/s 2021-05-12T12:20:44.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2021-05-12T12:20:44.136 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:20:49.157 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-05-12T12:20:49.157 INFO:tasks.workunit.client.0.smithi191.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2021-05-12T12:20:49.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2021-05-12T12:20:49.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:20:50.993 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:20:50.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:20:52.781 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:20:52.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2021-05-12T12:20:53.270 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records in 2021-05-12T12:20:53.271 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records out 2021-05-12T12:20:53.271 INFO:tasks.workunit.client.0.smithi191.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.485157 s, 2.2 GB/s 2021-05-12T12:20:53.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2021-05-12T12:20:53.280 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:02.053 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-05-12T12:21:02.053 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2021-05-12T12:21:02.054 INFO:tasks.workunit.client.0.smithi191.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2021-05-12T12:21:02.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2021-05-12T12:21:02.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:21:03.430 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:21:03.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-05-12T12:21:03.446 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:04.969 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-05-12T12:21:04.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2021-05-12T12:21:05.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:21:05.854 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:21:05.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2021-05-12T12:21:07.889 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:21:07.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-05-12T12:21:08.371 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records in 2021-05-12T12:21:08.372 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records out 2021-05-12T12:21:08.372 INFO:tasks.workunit.client.0.smithi191.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.478543 s, 2.2 GB/s 2021-05-12T12:21:08.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-05-12T12:21:08.382 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:08.387 INFO:tasks.workunit.client.0.smithi191.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2021-05-12T12:21:08.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2021-05-12T12:21:08.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:21:10.169 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:21:10.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-05-12T12:21:17.889 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records in 2021-05-12T12:21:17.889 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records out 2021-05-12T12:21:17.889 INFO:tasks.workunit.client.0.smithi191.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.58668 s, 142 MB/s 2021-05-12T12:21:17.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-05-12T12:21:17.901 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:19.672 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-05-12T12:21:19.672 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-05-12T12:21:19.672 INFO:tasks.workunit.client.0.smithi191.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2021-05-12T12:21:19.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:21:20.957 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:21:20.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2021-05-12T12:21:21.450 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records in 2021-05-12T12:21:21.451 INFO:tasks.workunit.client.0.smithi191.stderr:1024+0 records out 2021-05-12T12:21:21.451 INFO:tasks.workunit.client.0.smithi191.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.489267 s, 2.2 GB/s 2021-05-12T12:21:21.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2021-05-12T12:21:21.460 INFO:tasks.workunit.client.0.smithi191.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:30.233 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2021-05-12T12:21:30.233 INFO:tasks.workunit.client.0.smithi191.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-05-12T12:21:30.233 INFO:tasks.workunit.client.0.smithi191.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2021-05-12T12:21:30.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:21:31.524 INFO:tasks.workunit.client.0.smithi191.stdout:fsck success 2021-05-12T12:21:31.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-05-12T12:21:31.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:21:31.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:21:31.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:21:31.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:21:31.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T12:21:31.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:21:31.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:21:31.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:21:31.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:21:31.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T12:21:31.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T12:21:31.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:21:31.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:21:31.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:21:31.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:21:31.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:21:31.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:21:31.532 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:21:31.532 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:21:31.533 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:21:31.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:21:31.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:21:31.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:21:31.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:21:31.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:21:31.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:21:31.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:21:31.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:31.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:21:31.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:21:31.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T12:21:31.537 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:21:31.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:21:31.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:31.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-05-12T12:21:31.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:21:31.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:21:31.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:21:31.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:21:31.552 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:31.549+0000 7f93c458f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:31.555 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:31.552+0000 7f93c458f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:31.556 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:31.553+0000 7f93c458f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:31.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:21:31.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:21:31.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:21:31.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:21:31.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:21:31.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:31.903 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:21:31.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:21:31.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:31.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:32.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:32.574 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:32.571+0000 7f93c458f0c0 -1 Falling back to public interface 2021-05-12T12:21:33.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:33.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:33.258 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:21:33.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:21:33.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:33.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:33.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:34.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:34.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:34.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:21:34.627 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:21:34.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:34.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:34.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:35.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:35.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:35.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:21:35.999 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:21:36.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:36.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:36.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:37.141 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:37.138+0000 7f93c458f0c0 -1 osd.0 74 log_to_monitors {default=true} 2021-05-12T12:21:37.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:37.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:37.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:21:37.365 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:21:37.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:37.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:37.739 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 85 up_thru 85 down_at 75 last_clean_interval [62,74) [v2:127.0.0.1:6802/141537,v1:127.0.0.1:6803/141537] [v2:127.0.0.1:6804/141537,v1:127.0.0.1:6805/141537] exists,up d2d373bf-6a2c-4915-afa2-7895259f1e1d 2021-05-12T12:21:37.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:21:37.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:21:37.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:21:37.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T12:21:37.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=141537 2021-05-12T12:21:37.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-05-12T12:21:37.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:21:37.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:21:37.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:21:37.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:21:37.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T12:21:37.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:21:37.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:21:37.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:21:37.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:21:37.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T12:21:37.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T12:21:37.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:21:37.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:21:37.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:21:37.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:21:37.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:21:37.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:21:37.749 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:21:37.749 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:21:37.749 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:21:37.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:21:37.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:21:37.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:21:37.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:21:37.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:21:37.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:21:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:21:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:21:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:21:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T12:21:37.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:21:37.752 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:21:37.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:37.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-05-12T12:21:37.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:21:37.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:21:37.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:21:37.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:21:37.764 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:37.761+0000 7f74ff93a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:37.765 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:37.762+0000 7f74ff93a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:37.767 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:37.764+0000 7f74ff93a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:38.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:21:38.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:21:38.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:21:38.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:21:38.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:21:38.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:38.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:21:38.119 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:21:38.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:38.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:21:38.282 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:38.279+0000 7f74ff93a0c0 -1 Falling back to public interface 2021-05-12T12:21:38.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:39.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:39.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:39.520 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:21:39.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:21:39.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:39.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:21:39.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:40.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:40.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:40.891 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:21:40.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:21:40.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:40.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:21:41.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:41.746 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:41.743+0000 7f74ff93a0c0 -1 osd.1 77 log_to_monitors {default=true} 2021-05-12T12:21:42.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:42.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:42.256 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:21:42.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:21:42.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:42.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:21:42.722 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 78 last_clean_interval [65,77) [v2:127.0.0.1:6810/142534,v1:127.0.0.1:6811/142534] [v2:127.0.0.1:6812/142534,v1:127.0.0.1:6813/142534] exists,up f71f7391-473d-4cc3-8e29-43a620de60a1 2021-05-12T12:21:42.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:21:42.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:21:42.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:21:42.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T12:21:42.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=142534 2021-05-12T12:21:42.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-05-12T12:21:42.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:21:42.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:21:42.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:21:42.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:21:42.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T12:21:42.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:21:42.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:21:42.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:21:42.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:21:42.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T12:21:42.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T12:21:42.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:21:42.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:21:42.739 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:21:42.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:21:42.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:21:42.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:21:42.741 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:21:42.741 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:21:42.741 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:21:42.742 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:21:42.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:21:42.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:21:42.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:21:42.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:21:42.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:21:42.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:21:42.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:42.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:21:42.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:21:42.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T12:21:42.747 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:21:42.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:21:42.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:42.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-05-12T12:21:42.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:21:42.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:21:42.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:21:42.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:21:42.752 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:42.743+0000 7fd33c7bb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:42.752 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:42.744+0000 7fd33c7bb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:42.753 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:42.746+0000 7fd33c7bb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:43.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:21:43.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:21:43.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:21:43.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:21:43.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:21:43.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:43.126 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:21:43.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:21:43.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:43.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:21:43.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:43.522 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:43.519+0000 7fd33c7bb0c0 -1 Falling back to public interface 2021-05-12T12:21:44.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:44.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:44.481 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:21:44.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:21:44.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:44.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:21:44.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:45.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:45.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:45.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:21:45.846 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:21:45.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:45.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:21:46.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:47.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:47.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:47.213 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:21:47.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:21:47.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:47.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:21:47.249 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:47.246+0000 7fd33c7bb0c0 -1 osd.2 80 log_to_monitors {default=true} 2021-05-12T12:21:47.654 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 92 up_thru 78 down_at 81 last_clean_interval [68,80) [v2:127.0.0.1:6818/143485,v1:127.0.0.1:6819/143485] [v2:127.0.0.1:6820/143485,v1:127.0.0.1:6821/143485] exists,up ad18b5ee-7ebd-47ea-8856-c1d530f45233 2021-05-12T12:21:47.654 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:21:47.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:21:47.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:21:47.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T12:21:47.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=143485 2021-05-12T12:21:47.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-05-12T12:21:47.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:21:47.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:21:47.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T12:21:47.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:21:47.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T12:21:47.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:21:47.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:21:47.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:21:47.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:21:47.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T12:21:47.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T12:21:47.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:21:47.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:21:47.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:21:47.662 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:21:47.663 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:21:47.663 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:21:47.663 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:21:47.663 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:21:47.664 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:21:47.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:21:47.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' 2021-05-12T12:21:47.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:21:47.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:21:47.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:21:47.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:21:47.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:21:47.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:47.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:21:47.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:21:47.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T12:21:47.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T12:21:47.667 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.3 2021-05-12T12:21:47.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=b6fd7245-e17c-4f43-ab57-45760e0a3406 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.99184/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:47.668 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-05-12T12:21:47.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T12:21:47.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:21:47.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:21:47.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:21:47.678 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:47.675+0000 7f3b42aa60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:47.680 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:47.677+0000 7f3b42aa60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:47.682 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:47.679+0000 7f3b42aa60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:48.449 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:48.446+0000 7f3b42aa60c0 -1 Falling back to public interface 2021-05-12T12:21:48.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T12:21:48.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:21:48.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:21:48.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:21:48.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:21:48.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:48.526 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:21:48.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:21:48.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:48.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:21:49.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:50.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:50.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:50.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:21:50.217 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:21:50.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:50.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:21:50.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:51.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:51.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:51.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:21:51.589 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:21:51.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:51.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:21:51.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:52.243 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:21:52.240+0000 7f3b42aa60c0 -1 osd.3 82 log_to_monitors {default=true} 2021-05-12T12:21:52.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:52.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:52.950 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:21:52.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:21:52.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:52.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:21:53.422 INFO:tasks.workunit.client.0.smithi191.stdout:osd.3 up in weight 1 up_from 96 up_thru 96 down_at 83 last_clean_interval [72,82) [v2:127.0.0.1:6826/144446,v1:127.0.0.1:6827/144446] [v2:127.0.0.1:6828/144446,v1:127.0.0.1:6829/144446] exists,up 16258e95-8742-4373-96ca-e9d8bda153b0 2021-05-12T12:21:53.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:21:53.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:21:53.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:21:53.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T12:21:53.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=144446 2021-05-12T12:21:53.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T12:21:53.449 INFO:tasks.workunit.client.0.smithi191.stdout:hints = 1 2021-05-12T12:21:53.450 INFO:tasks.workunit.client.0.smithi191.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T12:21:53.450 INFO:tasks.workunit.client.0.smithi191.stdout:Object prefix: benchmark_data_smithi191_145392 2021-05-12T12:21:53.450 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:21:53.450 INFO:tasks.workunit.client.0.smithi191.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:21:54.450 INFO:tasks.workunit.client.0.smithi191.stdout: 1 16 43 27 0.105461 0.105469 0.182565 0.264515 2021-05-12T12:21:55.450 INFO:tasks.workunit.client.0.smithi191.stdout: 2 16 53 37 0.0722575 0.0390625 0.0782491 0.247584 2021-05-12T12:21:56.450 INFO:tasks.workunit.client.0.smithi191.stdout: 3 16 53 37 0.0481707 0 - 0.247584 2021-05-12T12:21:57.450 INFO:tasks.workunit.client.0.smithi191.stdout: 4 16 53 37 0.0361285 0 - 0.247584 2021-05-12T12:21:58.450 INFO:tasks.workunit.client.0.smithi191.stdout: 5 16 53 37 0.0289027 0 - 0.247584 2021-05-12T12:21:59.450 INFO:tasks.workunit.client.0.smithi191.stdout: 6 16 53 37 0.0240855 0 - 0.247584 2021-05-12T12:22:00.450 INFO:tasks.workunit.client.0.smithi191.stdout: 7 16 53 37 0.0206446 0 - 0.247584 2021-05-12T12:22:01.451 INFO:tasks.workunit.client.0.smithi191.stdout: 8 16 53 37 0.018064 0 - 0.247584 2021-05-12T12:22:02.451 INFO:tasks.workunit.client.0.smithi191.stdout: 9 16 53 37 0.0160569 0 - 0.247584 2021-05-12T12:22:03.451 INFO:tasks.workunit.client.0.smithi191.stdout: 10 16 53 37 0.0144512 0 - 0.247584 2021-05-12T12:22:04.451 INFO:tasks.workunit.client.0.smithi191.stdout: 11 16 72 56 0.0198837 0.00824653 0.23184 1.82178 2021-05-12T12:22:05.451 INFO:tasks.workunit.client.0.smithi191.stdout: 12 16 93 77 0.0250617 0.0820312 0.152993 1.39497 2021-05-12T12:22:06.451 INFO:tasks.workunit.client.0.smithi191.stdout: 13 16 114 98 0.0294432 0.0820312 0.183886 1.13088 2021-05-12T12:22:07.452 INFO:tasks.workunit.client.0.smithi191.stdout: 14 16 161 145 0.0404522 0.183594 0.134684 1.52645 2021-05-12T12:22:08.452 INFO:tasks.workunit.client.0.smithi191.stdout: 15 16 235 219 0.0570237 0.289062 0.114782 1.08566 2021-05-12T12:22:09.452 INFO:tasks.workunit.client.0.smithi191.stdout: 16 16 309 293 0.0715237 0.289062 0.262478 0.869078 2021-05-12T12:22:10.452 INFO:tasks.workunit.client.0.smithi191.stdout: 17 16 360 344 0.0790335 0.199219 0.455523 0.786645 2021-05-12T12:22:11.452 INFO:tasks.workunit.client.0.smithi191.stdout: 18 16 433 417 0.0904828 0.285156 0.208996 0.689945 2021-05-12T12:22:12.452 INFO:tasks.workunit.client.0.smithi191.stdout: 19 16 502 486 0.0999047 0.269531 0.245549 0.622781 2021-05-12T12:22:13.452 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T12:22:13.450419+0000 min lat: 0.0668932 max lat: 13.4878 avg lat: 0.586919 2021-05-12T12:22:13.453 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:22:13.453 INFO:tasks.workunit.client.0.smithi191.stdout: 20 16 556 540 0.105455 0.210938 0.297007 0.586919 2021-05-12T12:22:14.452 INFO:tasks.workunit.client.0.smithi191.stdout: 21 16 629 613 0.11401 0.285156 0.308538 0.544799 2021-05-12T12:22:15.453 INFO:tasks.workunit.client.0.smithi191.stdout: 22 16 701 685 0.12161 0.28125 0.159241 0.511251 2021-05-12T12:22:16.453 INFO:tasks.workunit.client.0.smithi191.stdout: 23 16 770 754 0.12804 0.269531 0.136892 0.485876 2021-05-12T12:22:17.453 INFO:tasks.workunit.client.0.smithi191.stdout: 24 16 823 807 0.13133 0.207031 0.141783 0.473016 2021-05-12T12:22:18.453 INFO:tasks.workunit.client.0.smithi191.stdout: 25 16 900 884 0.138106 0.300781 0.258873 0.449877 2021-05-12T12:22:19.453 INFO:tasks.workunit.client.0.smithi191.stdout: 26 16 963 947 0.142259 0.246094 0.253987 0.437575 2021-05-12T12:22:20.453 INFO:tasks.workunit.client.0.smithi191.stdout: 27 16 1033 1017 0.147116 0.273438 0.258905 0.423731 2021-05-12T12:22:21.453 INFO:tasks.workunit.client.0.smithi191.stdout: 28 16 1087 1071 0.149395 0.210938 0.200287 0.415998 2021-05-12T12:22:22.453 INFO:tasks.workunit.client.0.smithi191.stdout: 29 16 1167 1151 0.155018 0.3125 0.112251 0.402408 2021-05-12T12:22:23.453 INFO:tasks.workunit.client.0.smithi191.stdout: 30 14 1236 1222 0.159094 0.277344 0.108688 0.391009 2021-05-12T12:22:24.454 INFO:tasks.workunit.client.0.smithi191.stdout:Total time run: 30.0937 2021-05-12T12:22:24.454 INFO:tasks.workunit.client.0.smithi191.stdout:Total writes made: 1236 2021-05-12T12:22:24.455 INFO:tasks.workunit.client.0.smithi191.stdout:Write size: 4096 2021-05-12T12:22:24.455 INFO:tasks.workunit.client.0.smithi191.stdout:Object size: 4096 2021-05-12T12:22:24.455 INFO:tasks.workunit.client.0.smithi191.stdout:Bandwidth (MB/sec): 0.160436 2021-05-12T12:22:24.455 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Bandwidth: 0.124189 2021-05-12T12:22:24.456 INFO:tasks.workunit.client.0.smithi191.stdout:Max bandwidth (MB/sec): 0.3125 2021-05-12T12:22:24.456 INFO:tasks.workunit.client.0.smithi191.stdout:Min bandwidth (MB/sec): 0 2021-05-12T12:22:24.456 INFO:tasks.workunit.client.0.smithi191.stdout:Average IOPS: 41 2021-05-12T12:22:24.456 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev IOPS: 31.7969 2021-05-12T12:22:24.456 INFO:tasks.workunit.client.0.smithi191.stdout:Max IOPS: 80 2021-05-12T12:22:24.457 INFO:tasks.workunit.client.0.smithi191.stdout:Min IOPS: 0 2021-05-12T12:22:24.457 INFO:tasks.workunit.client.0.smithi191.stdout:Average Latency(s): 0.388945 2021-05-12T12:22:24.457 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Latency(s): 1.27685 2021-05-12T12:22:24.457 INFO:tasks.workunit.client.0.smithi191.stdout:Max latency(s): 13.4878 2021-05-12T12:22:24.458 INFO:tasks.workunit.client.0.smithi191.stdout:Min latency(s): 0.0668932 2021-05-12T12:22:24.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2021-05-12T12:22:24.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:22:24.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:22:24.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:22:24.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:22:24.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:22:24.574 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:22:24.574 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:22:24.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:22:24.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:22:24.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:22:24.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:22:24.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:22:24.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:22:24.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:22:24.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:22:25.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:22:25.033 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:22:25.033 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:22:25.034 INFO:tasks.workunit.client.0.smithi191.stderr:3' 2021-05-12T12:22:25.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:22:25.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:22:25.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:22:25.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=365072220172 2021-05-12T12:22:25.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 365072220172 2021-05-12T12:22:25.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-365072220172' 2021-05-12T12:22:25.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:22:25.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:22:25.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=377957122059 2021-05-12T12:22:25.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 377957122059 2021-05-12T12:22:25.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-365072220172 1-377957122059' 2021-05-12T12:22:25.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:22:25.289 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:22:25.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=395136991242 2021-05-12T12:22:25.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 395136991242 2021-05-12T12:22:25.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-365072220172 1-377957122059 2-395136991242' 2021-05-12T12:22:25.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:22:25.416 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:22:25.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=412316860425 2021-05-12T12:22:25.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 412316860425 2021-05-12T12:22:25.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-365072220172 1-377957122059 2-395136991242 3-412316860425' 2021-05-12T12:22:25.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:22:25.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-365072220172 2021-05-12T12:22:25.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:22:25.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:22:25.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-365072220172 2021-05-12T12:22:25.546 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:22:25.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=365072220172 2021-05-12T12:22:25.547 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 365072220172 2021-05-12T12:22:25.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 365072220172' 2021-05-12T12:22:25.547 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:22:25.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 365072220170 -lt 365072220172 2021-05-12T12:22:25.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:22:26.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:22:26.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:22:27.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 365072220172 -lt 365072220172 2021-05-12T12:22:27.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:22:27.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-377957122059 2021-05-12T12:22:27.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:22:27.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:22:27.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-377957122059 2021-05-12T12:22:27.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:22:27.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=377957122059 2021-05-12T12:22:27.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 377957122059' 2021-05-12T12:22:27.277 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 377957122059 2021-05-12T12:22:27.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:22:27.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 377957122059 -lt 377957122059 2021-05-12T12:22:27.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:22:27.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-395136991242 2021-05-12T12:22:27.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:22:27.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:22:27.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-395136991242 2021-05-12T12:22:27.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:22:27.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=395136991242 2021-05-12T12:22:27.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 395136991242' 2021-05-12T12:22:27.640 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 395136991242 2021-05-12T12:22:27.640 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:22:27.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 395136991242 -lt 395136991242 2021-05-12T12:22:27.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:22:27.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-412316860425 2021-05-12T12:22:27.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:22:27.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:22:28.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-412316860425 2021-05-12T12:22:28.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:22:28.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=412316860425 2021-05-12T12:22:28.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 412316860425' 2021-05-12T12:22:28.002 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 412316860425 2021-05-12T12:22:28.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:22:28.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 412316860425 -lt 412316860425 2021-05-12T12:22:28.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:22:28.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:22:28.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:22:28.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 16 == 0 2021-05-12T12:22:28.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:22:28.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:22:28.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:22:28.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:22:28.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:22:28.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:22:28.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:22:29.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=16 2021-05-12T12:22:29.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:22:29.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:22:29.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:22:29.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 16 = 16 2021-05-12T12:22:29.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:22:29.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:22:29.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:25: run: teardown td/osd-bluefs-volume-ops 2021-05-12T12:22:29.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:22:29.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:29.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-05-12T12:22:29.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:29.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:29.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:29.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:29.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:29.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:29.788 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:29.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:29.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:29.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:29.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:29.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:29.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:29.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:29.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:29.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:29.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:29.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:29.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-05-12T12:22:30.152 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:30.152 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:30.153 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:22:30.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.99184 2021-05-12T12:22:30.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:30.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:30.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:22:30.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-bluefs-volume-ops 0 2021-05-12T12:22:30.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:22:30.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:22:30.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-05-12T12:22:30.156 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:30.156 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:30.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:30.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:30.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:30.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:30.158 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:30.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:30.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:30.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:30.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:30.160 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:30.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:30.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:30.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:30.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:30.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:30.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:22:30.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-05-12T12:22:30.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:30.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:30.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99184 2021-05-12T12:22:30.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.99184 2021-05-12T12:22:30.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:30.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:30.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:22:30.166 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:22:30.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:22:30.238 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2021-05-12T12:22:30.238 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2021-05-12T12:22:30.298 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:22:30.305 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:22:30.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:22:30.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:22:30.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:22:30.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:22:30.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:22:30.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:22:30.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:22:30.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:22:30.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-config 2021-05-12T12:22:30.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2021-05-12T12:22:30.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2021-05-12T12:22:30.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2021-05-12T12:22:30.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2021-05-12T12:22:30.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2021-05-12T12:22:30.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2021-05-12T12:22:30.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=49b45dd7-13f3-4af3-a9ca-34322f339ce9 --auth-supported=none ' 2021-05-12T12:22:30.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2021-05-12T12:22:30.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2021-05-12T12:22:30.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:22:30.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2021-05-12T12:22:30.313 INFO:tasks.workunit.client.0.smithi191.stderr:TEST_config_track 2021-05-12T12:22:30.313 INFO:tasks.workunit.client.0.smithi191.stderr:TEST_default_adjustment' 2021-05-12T12:22:30.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-05-12T12:22:30.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-05-12T12:22:30.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-05-12T12:22:30.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-05-12T12:22:30.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:30.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:30.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:30.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:30.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:30.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:30.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:30.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:30.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:30.316 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:30.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:30.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:30.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:30.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:30.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:30.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:30.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:30.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:30.318 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:30.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:30.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:30.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:30.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:30.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:30.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:30.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145958 2021-05-12T12:22:30.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:30.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:30.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-05-12T12:22:30.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:22:30.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:30.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:30.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.145958 2021-05-12T12:22:30.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:22:30.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:22:30.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:22:30.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:22:30.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-05-12T12:22:30.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2021-05-12T12:22:30.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2021-05-12T12:22:30.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2021-05-12T12:22:30.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-05-12T12:22:30.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:22:30.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:22:30.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:22:30.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-05-12T12:22:30.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-05-12T12:22:30.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:22:30.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:30.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:30.681 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:30.681 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:30.681 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:30.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145958/$cluster-$name.asok' 2021-05-12T12:22:30.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:22:30.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:22:30.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:22:30.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:30.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:30.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:22:30.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:22:30.910 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:30.910 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:30.910 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:30.910 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:30.910 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:30.911 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:30.911 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.145958/ceph-mon.a.asok 2021-05-12T12:22:30.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:30.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.145958/ceph-mon.a.asok config get fsid 2021-05-12T12:22:31.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:22:31.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:31.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:31.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:22:31.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:22:31.004 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:31.005 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:31.005 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:31.005 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:31.005 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:31.005 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:31.006 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.145958/ceph-mon.a.asok 2021-05-12T12:22:31.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:31.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.145958/ceph-mon.a.asok config get mon_host 2021-05-12T12:22:31.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2021-05-12T12:22:31.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2021-05-12T12:22:31.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:22:31.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:22:31.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:22:31.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2021-05-12T12:22:31.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:22:31.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:22:31.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:31.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:31.969 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:31.969 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:31.969 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:31.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145958/$cluster-$name.asok' 2021-05-12T12:22:31.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:31.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:31.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2021-05-12T12:22:31.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2021-05-12T12:22:31.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-05-12T12:22:31.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2021-05-12T12:22:31.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:22:31.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:22:31.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:22:31.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2021-05-12T12:22:31.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=49b45dd7-13f3-4af3-a9ca-34322f339ce9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-05-12T12:22:31.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:22:31.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:22:31.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:22:31.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-05-12T12:22:31.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-05-12T12:22:31.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:22:31.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:22:31.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-05-12T12:22:31.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:22:31.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:31.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:31.999 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:31.999 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:31.999 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:31.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145958/$cluster-$name.asok' 2021-05-12T12:22:31.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' 2021-05-12T12:22:32.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:22:32.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:22:32.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:22:32.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-05-12T12:22:32.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-05-12T12:22:32.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:22:32.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:22:32.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:32.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:22:32.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2021-05-12T12:22:32.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-config/0 2021-05-12T12:22:32.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:22:32.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=75c97a4d-0eec-4b97-9a28-97cb23eb7945 2021-05-12T12:22:32.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 75c97a4d-0eec-4b97-9a28-97cb23eb7945' 2021-05-12T12:22:32.003 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 75c97a4d-0eec-4b97-9a28-97cb23eb7945 2021-05-12T12:22:32.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:22:32.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCIyJtg7Uo+ABAAtM1MeF5LYjsM58KWJQTYpw== 2021-05-12T12:22:32.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCIyJtg7Uo+ABAAtM1MeF5LYjsM58KWJQTYpw=="}' 2021-05-12T12:22:32.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 75c97a4d-0eec-4b97-9a28-97cb23eb7945 -i td/osd-config/0/new.json 2021-05-12T12:22:32.333 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:22:32.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-config/0/new.json 2021-05-12T12:22:32.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=49b45dd7-13f3-4af3-a9ca-34322f339ce9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQCIyJtg7Uo+ABAAtM1MeF5LYjsM58KWJQTYpw== --osd-uuid 75c97a4d-0eec-4b97-9a28-97cb23eb7945 2021-05-12T12:22:32.365 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:32.362+0000 7fa036f8c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:32.367 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:32.364+0000 7fa036f8c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:32.367 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:32.365+0000 7fa036f8c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:32.368 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:32.365+0000 7fa036f8c0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:32.369 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:32.365+0000 7fa036f8c0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:32.369 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:32.365+0000 7fa036f8c0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:32.405 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:32.402+0000 7fa036f8c0c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-05-12T12:22:36.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-config/0/keyring 2021-05-12T12:22:36.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:22:36.954 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:22:36.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:22:36.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:22:37.464 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:22:37.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:22:37.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=49b45dd7-13f3-4af3-a9ca-34322f339ce9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-05-12T12:22:37.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:22:37.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:22:37.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:22:37.485 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:37.482+0000 7f465fc0f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:37.487 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:37.484+0000 7f465fc0f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:37.488 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:37.485+0000 7f465fc0f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:37.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:22:37.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:22:37.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:22:37.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:22:37.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:22:37.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:37.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:22:37.829 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:22:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:37.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:22:38.009 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:38.005+0000 7f465fc0f0c0 -1 Falling back to public interface 2021-05-12T12:22:38.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:39.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:39.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:39.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:22:39.218 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:22:39.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:39.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:22:39.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:40.187 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:40.184+0000 7f465fc0f0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:22:40.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:40.584 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:40.584 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:22:40.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:22:40.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:40.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:22:40.952 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/147078,v1:127.0.0.1:6803/147078] [v2:127.0.0.1:6804/147078,v1:127.0.0.1:6805/147078] exists,up 75c97a4d-0eec-4b97-9a28-97cb23eb7945 2021-05-12T12:22:40.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:22:40.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:22:40.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:22:40.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2021-05-12T12:22:40.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:22:40.955 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:22:40.955 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:40.955 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:40.955 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:40.956 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.145958/ceph-osd.0.asok 2021-05-12T12:22:40.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2021-05-12T12:22:40.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.145958/ceph-osd.0.asok log flush 2021-05-12T12:22:41.046 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:22:41.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-05-12T12:22:41.055 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T12:22:40.186+0000 7f4657399700 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2021-05-12T12:22:41.055 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T12:22:40.487+0000 7f465fc0f0c0 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2021-05-12T12:22:41.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-05-12T12:22:41.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:41.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:41.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:41.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:41.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:41.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:41.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:41.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:41.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:41.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:41.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:41.228 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:41.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:41.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:41.229 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:41.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:41.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:41.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:41.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:41.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:41.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:41.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:41.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145958 2021-05-12T12:22:41.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:41.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:41.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-05-12T12:22:41.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-05-12T12:22:41.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-05-12T12:22:41.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-05-12T12:22:41.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:41.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:41.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:41.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:41.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:41.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:41.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:41.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:41.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:41.242 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:41.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:41.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:41.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:41.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:41.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:41.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:41.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:41.244 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:41.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:41.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:41.246 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:41.246 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.246 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:41.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145958 2021-05-12T12:22:41.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:41.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:41.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-05-12T12:22:41.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:22:41.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:41.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.145958 2021-05-12T12:22:41.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:22:41.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:22:41.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:22:41.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-05-12T12:22:41.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2021-05-12T12:22:41.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2021-05-12T12:22:41.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2021-05-12T12:22:41.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-05-12T12:22:41.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:22:41.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:22:41.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:22:41.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-05-12T12:22:41.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-05-12T12:22:41.598 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:22:41.598 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:41.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:41.599 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:41.599 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.599 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:41.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145958/$cluster-$name.asok' 2021-05-12T12:22:41.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:22:41.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:22:41.817 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:22:41.818 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:41.818 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:41.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:22:41.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:22:41.819 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:41.819 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:41.820 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:41.820 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:41.820 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.820 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:41.820 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.145958/ceph-mon.a.asok 2021-05-12T12:22:41.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:41.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.145958/ceph-mon.a.asok config get fsid 2021-05-12T12:22:41.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:22:41.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:41.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:41.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:22:41.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:22:41.914 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:41.914 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:41.915 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:41.915 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:41.915 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.915 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:41.915 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.145958/ceph-mon.a.asok 2021-05-12T12:22:41.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:41.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.145958/ceph-mon.a.asok config get mon_host 2021-05-12T12:22:42.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2021-05-12T12:22:42.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2021-05-12T12:22:42.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:22:42.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:22:42.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:22:42.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2021-05-12T12:22:42.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:22:42.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:22:42.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:42.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:42.888 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:42.888 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:42.888 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:42.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145958/$cluster-$name.asok' 2021-05-12T12:22:42.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:42.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:42.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2021-05-12T12:22:42.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2021-05-12T12:22:42.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:22:42.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:22:42.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:22:42.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2021-05-12T12:22:42.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=49b45dd7-13f3-4af3-a9ca-34322f339ce9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-05-12T12:22:42.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:22:42.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:22:42.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:22:42.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-05-12T12:22:42.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-05-12T12:22:42.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:22:42.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:22:42.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-05-12T12:22:42.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:22:42.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:42.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:42.916 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:42.916 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:42.916 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:42.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145958/$cluster-$name.asok' 2021-05-12T12:22:42.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' 2021-05-12T12:22:42.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:22:42.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:22:42.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:22:42.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-05-12T12:22:42.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-05-12T12:22:42.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:22:42.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:22:42.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:42.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:22:42.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:22:42.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-config/0 2021-05-12T12:22:42.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:22:42.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=133750f1-c298-48ca-a14e-77c8daa7c76e 2021-05-12T12:22:42.920 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 133750f1-c298-48ca-a14e-77c8daa7c76e 2021-05-12T12:22:42.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 133750f1-c298-48ca-a14e-77c8daa7c76e' 2021-05-12T12:22:42.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:22:42.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCSyJtghsj1NhAASOPEvKNTUyUL8OrGe7/VAg== 2021-05-12T12:22:42.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCSyJtghsj1NhAASOPEvKNTUyUL8OrGe7/VAg=="}' 2021-05-12T12:22:42.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 133750f1-c298-48ca-a14e-77c8daa7c76e -i td/osd-config/0/new.json 2021-05-12T12:22:43.240 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:22:43.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-config/0/new.json 2021-05-12T12:22:43.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=49b45dd7-13f3-4af3-a9ca-34322f339ce9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCSyJtghsj1NhAASOPEvKNTUyUL8OrGe7/VAg== --osd-uuid 133750f1-c298-48ca-a14e-77c8daa7c76e 2021-05-12T12:22:43.273 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:43.270+0000 7fdb2f0ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:43.275 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:43.272+0000 7fdb2f0ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:43.276 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:43.273+0000 7fdb2f0ef0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:43.277 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:43.274+0000 7fdb2f0ef0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:43.278 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:43.274+0000 7fdb2f0ef0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:43.278 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:43.274+0000 7fdb2f0ef0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:43.315 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:43.312+0000 7fdb2f0ef0c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-05-12T12:22:47.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-config/0/keyring 2021-05-12T12:22:47.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:22:47.854 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:22:47.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:22:47.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:22:48.327 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:22:48.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:22:48.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=49b45dd7-13f3-4af3-a9ca-34322f339ce9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.145958/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:48.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:22:48.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:22:48.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:22:48.346 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:48.344+0000 7f86a5d790c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:48.348 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:48.345+0000 7f86a5d790c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:48.350 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:48.347+0000 7f86a5d790c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:48.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:22:48.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:22:48.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:22:48.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:22:48.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:22:48.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:48.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:22:48.686 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:22:48.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:48.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:22:48.868 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:48.865+0000 7f86a5d790c0 -1 Falling back to public interface 2021-05-12T12:22:49.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:50.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:50.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:50.043 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:22:50.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:22:50.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:50.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:22:50.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:51.083 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:51.079+0000 7f86a5d790c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:22:51.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:51.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:51.406 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:22:51.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:22:51.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:51.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:22:51.768 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/148761,v1:127.0.0.1:6803/148761] [v2:127.0.0.1:6804/148761,v1:127.0.0.1:6805/148761] exists,up 133750f1-c298-48ca-a14e-77c8daa7c76e 2021-05-12T12:22:51.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:22:51.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:22:51.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:22:51.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2021-05-12T12:22:51.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2021-05-12T12:22:51.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2021-05-12T12:22:51.779 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2021-05-12T12:22:51.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2021-05-12T12:22:51.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2021-05-12T12:22:51.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-05-12T12:22:51.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2021-05-12T12:22:51.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2021-05-12T12:22:51.909 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:22:51.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2021-05-12T12:22:51.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:22:51.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:22:51.920 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:51.920 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:51.921 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:51.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.145958/ceph-osd.0.asok 2021-05-12T12:22:51.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2021-05-12T12:22:51.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.145958/ceph-osd.0.asok log flush 2021-05-12T12:22:52.013 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:22:52.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-05-12T12:22:52.022 INFO:tasks.workunit.client.0.smithi191.stdout:2021-05-12T12:22:51.893+0000 7f869f507700 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2021-05-12T12:22:52.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2021-05-12T12:22:52.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2021-05-12T12:22:52.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:22:52.026 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:22:52.026 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:52.026 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.027 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:52.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.145958/ceph-osd.0.asok 2021-05-12T12:22:52.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2021-05-12T12:22:52.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.145958/ceph-osd.0.asok log reopen 2021-05-12T12:22:52.118 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:22:52.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-05-12T12:22:52.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:52.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:52.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:52.128 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:52.128 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:52.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:52.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:52.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:52.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:52.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:52.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:52.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:52.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:52.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:52.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:52.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:52.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:52.237 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:52.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:52.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:52.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:52.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.241 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:52.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145958 2021-05-12T12:22:52.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:52.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:52.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-05-12T12:22:52.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-05-12T12:22:52.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-05-12T12:22:52.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-05-12T12:22:52.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:52.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:52.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:52.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:52.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:52.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:52.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:52.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:52.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:52.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:52.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:52.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:52.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:52.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:52.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:52.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:52.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:52.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:52.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:52.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:52.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:52.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:52.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145958 2021-05-12T12:22:52.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:52.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:52.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-05-12T12:22:52.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:22:52.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:52.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.145958 2021-05-12T12:22:52.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:22:52.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:22:52.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:22:52.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-05-12T12:22:52.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2021-05-12T12:22:52.256 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2021-05-12T12:22:52.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2021-05-12T12:22:52.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2021-05-12T12:22:52.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2021-05-12T12:22:52.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2021-05-12T12:22:52.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2021-05-12T12:22:52.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2021-05-12T12:22:52.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2021-05-12T12:22:52.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2021-05-12T12:22:52.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2021-05-12T12:22:52.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2021-05-12T12:22:52.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2021-05-12T12:22:52.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2021-05-12T12:22:52.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2021-05-12T12:22:52.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2021-05-12T12:22:52.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2021-05-12T12:22:52.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2021-05-12T12:22:52.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2021-05-12T12:22:52.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-05-12T12:22:52.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:52.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:52.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:52.335 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:52.335 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:52.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:52.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:52.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:52.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:52.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:52.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:52.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:52.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:52.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:52.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:52.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:52.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:52.340 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:52.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:52.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:52.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:52.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:52.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145958 2021-05-12T12:22:52.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:52.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:52.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:22:52.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-config 0 2021-05-12T12:22:52.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:52.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:22:52.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:52.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:52.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:52.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:52.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:52.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:52.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:52.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:52.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:52.348 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:52.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:52.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:52.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:52.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:52.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:52.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:52.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:22:52.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:52.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:52.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145958 2021-05-12T12:22:52.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145958 2021-05-12T12:22:52.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:52.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:52.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:22:52.354 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:22:52.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:22:52.416 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2021-05-12T12:22:52.417 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2021-05-12T12:22:52.477 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:22:52.484 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:22:52.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:22:52.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:22:52.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:22:52.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:22:52.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:22:52.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:22:52.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:22:52.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:22:52.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-copy-from 2021-05-12T12:22:52.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2021-05-12T12:22:52.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2021-05-12T12:22:52.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2021-05-12T12:22:52.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2021-05-12T12:22:52.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2021-05-12T12:22:52.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2021-05-12T12:22:52.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=73d88e8f-3aaf-44a0-89c4-5dae4a1685b5 --auth-supported=none ' 2021-05-12T12:22:52.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2021-05-12T12:22:52.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2021-05-12T12:22:52.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:22:52.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2021-05-12T12:22:52.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2021-05-12T12:22:52.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2021-05-12T12:22:52.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2021-05-12T12:22:52.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2021-05-12T12:22:52.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-05-12T12:22:52.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:52.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-05-12T12:22:52.492 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:52.493 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:52.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:52.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:52.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:52.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:52.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:52.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:52.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:52.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:52.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:52.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:52.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:52.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:52.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:52.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:52.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-05-12T12:22:52.500 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:52.500 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.500 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:22:52.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.149458 2021-05-12T12:22:52.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:52.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:52.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2021-05-12T12:22:52.501 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:22:52.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:22:52.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.149458 2021-05-12T12:22:52.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:22:52.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:22:52.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:22:52.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:22:52.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2021-05-12T12:22:52.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2021-05-12T12:22:52.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2021-05-12T12:22:52.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2021-05-12T12:22:52.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2021-05-12T12:22:52.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:22:52.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:22:52.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:22:52.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2021-05-12T12:22:52.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2021-05-12T12:22:52.850 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:22:52.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:52.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:52.852 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:52.852 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.853 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:22:52.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.149458/$cluster-$name.asok' 2021-05-12T12:22:52.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.149458/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:22:53.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:22:53.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:22:53.106 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:53.106 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:53.107 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:22:53.107 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:22:53.107 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:53.108 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:53.108 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:53.109 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:53.109 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:53.109 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:22:53.110 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.149458/ceph-mon.a.asok 2021-05-12T12:22:53.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:53.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.149458/ceph-mon.a.asok config get fsid 2021-05-12T12:22:53.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:22:53.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:53.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:53.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:22:53.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:22:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:53.200 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:53.201 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:22:53.201 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.149458/ceph-mon.a.asok 2021-05-12T12:22:53.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:53.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.149458/ceph-mon.a.asok config get mon_host 2021-05-12T12:22:53.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2021-05-12T12:22:53.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2021-05-12T12:22:53.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:22:53.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:22:53.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:22:53.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2021-05-12T12:22:53.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:22:54.154 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:22:54.155 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:54.155 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:54.155 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:54.155 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:54.155 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:22:54.156 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.149458/$cluster-$name.asok' 2021-05-12T12:22:54.156 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:54.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.149458/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:54.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2021-05-12T12:22:54.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2021-05-12T12:22:54.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:22:54.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:22:54.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:22:54.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2021-05-12T12:22:54.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=73d88e8f-3aaf-44a0-89c4-5dae4a1685b5 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-05-12T12:22:54.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:22:54.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:22:54.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:22:54.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2021-05-12T12:22:54.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2021-05-12T12:22:54.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:22:54.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:22:54.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-05-12T12:22:54.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:22:54.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:54.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:54.183 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:54.183 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:54.183 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:22:54.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.149458/$cluster-$name.asok' 2021-05-12T12:22:54.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.149458/$cluster-$name.asok' 2021-05-12T12:22:54.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:22:54.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:22:54.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:22:54.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-05-12T12:22:54.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-05-12T12:22:54.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:22:54.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:22:54.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:54.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:22:54.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:22:54.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-copy-from/0 2021-05-12T12:22:54.186 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:22:54.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=9899a4f1-e66a-4570-9aad-4c078502af94 2021-05-12T12:22:54.187 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 9899a4f1-e66a-4570-9aad-4c078502af94 2021-05-12T12:22:54.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 9899a4f1-e66a-4570-9aad-4c078502af94' 2021-05-12T12:22:54.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:22:54.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCeyJtgzNtLCxAAymf1lvBliYUfoLq+AvHhrA== 2021-05-12T12:22:54.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCeyJtgzNtLCxAAymf1lvBliYUfoLq+AvHhrA=="}' 2021-05-12T12:22:54.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 9899a4f1-e66a-4570-9aad-4c078502af94 -i td/osd-copy-from/0/new.json 2021-05-12T12:22:54.509 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:22:54.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-copy-from/0/new.json 2021-05-12T12:22:54.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=73d88e8f-3aaf-44a0-89c4-5dae4a1685b5 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.149458/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCeyJtgzNtLCxAAymf1lvBliYUfoLq+AvHhrA== --osd-uuid 9899a4f1-e66a-4570-9aad-4c078502af94 2021-05-12T12:22:54.541 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:54.538+0000 7f18b43410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:54.543 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:54.540+0000 7f18b43410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:54.544 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:54.541+0000 7f18b43410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:54.545 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:54.542+0000 7f18b43410c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-05-12T12:22:54.545 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:54.542+0000 7f18b43410c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-05-12T12:22:54.545 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:54.542+0000 7f18b43410c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-05-12T12:22:54.575 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:54.572+0000 7f18b43410c0 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2021-05-12T12:22:59.142 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-copy-from/0/keyring 2021-05-12T12:22:59.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:22:59.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:22:59.144 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:22:59.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:22:59.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:22:59.621 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:22:59.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:22:59.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=73d88e8f-3aaf-44a0-89c4-5dae4a1685b5 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.149458/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:59.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:22:59.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:22:59.639 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:59.636+0000 7f25ba3670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:59.641 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:59.638+0000 7f25ba3670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:59.643 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:22:59.639+0000 7f25ba3670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:59.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:22:59.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:22:59.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:22:59.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:22:59.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:22:59.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:59.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:22:59.980 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:22:59.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:59.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:00.158 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:00.155+0000 7f25ba3670c0 -1 Falling back to public interface 2021-05-12T12:23:00.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:01.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:01.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:01.336 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:23:01.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:01.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:01.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:01.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:02.333 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:02.330+0000 7f25ba3670c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:23:02.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:02.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:02.705 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:23:02.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:23:02.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:02.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:03.082 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/150578,v1:127.0.0.1:6803/150578] [v2:127.0.0.1:6804/150578,v1:127.0.0.1:6805/150578] exists,up 9899a4f1-e66a-4570-9aad-4c078502af94 2021-05-12T12:23:03.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:03.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:03.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:03.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2021-05-12T12:23:03.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2021-05-12T12:23:03.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:23:03.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:23:03.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:23:03.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2021-05-12T12:23:03.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=73d88e8f-3aaf-44a0-89c4-5dae4a1685b5 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-05-12T12:23:03.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:03.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:03.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:03.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2021-05-12T12:23:03.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2021-05-12T12:23:03.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:23:03.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:23:03.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-05-12T12:23:03.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:23:03.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:03.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:03.091 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:03.092 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:03.092 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:23:03.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.149458/$cluster-$name.asok' 2021-05-12T12:23:03.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.149458/$cluster-$name.asok' 2021-05-12T12:23:03.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:03.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:23:03.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:23:03.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-05-12T12:23:03.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-05-12T12:23:03.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:03.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:03.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:03.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:23:03.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:23:03.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-copy-from/1 2021-05-12T12:23:03.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:23:03.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e295ad2b-de5d-4739-940c-fa9c524ec04d 2021-05-12T12:23:03.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 e295ad2b-de5d-4739-940c-fa9c524ec04d' 2021-05-12T12:23:03.096 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 e295ad2b-de5d-4739-940c-fa9c524ec04d 2021-05-12T12:23:03.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:23:03.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCnyJtgZAWsBRAA8qV9jFpI/LkYxPjaUS0tuA== 2021-05-12T12:23:03.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCnyJtgZAWsBRAA8qV9jFpI/LkYxPjaUS0tuA=="}' 2021-05-12T12:23:03.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e295ad2b-de5d-4739-940c-fa9c524ec04d -i td/osd-copy-from/1/new.json 2021-05-12T12:23:03.577 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:23:03.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-copy-from/1/new.json 2021-05-12T12:23:03.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=73d88e8f-3aaf-44a0-89c4-5dae4a1685b5 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.149458/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCnyJtgZAWsBRAA8qV9jFpI/LkYxPjaUS0tuA== --osd-uuid e295ad2b-de5d-4739-940c-fa9c524ec04d 2021-05-12T12:23:03.609 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:03.606+0000 7f5559af00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:03.611 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:03.608+0000 7f5559af00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:03.611 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:03.608+0000 7f5559af00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:03.612 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:03.609+0000 7f5559af00c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-05-12T12:23:03.612 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:03.609+0000 7f5559af00c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-05-12T12:23:03.613 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:03.609+0000 7f5559af00c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-05-12T12:23:03.663 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:03.660+0000 7f5559af00c0 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2021-05-12T12:23:08.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-copy-from/1/keyring 2021-05-12T12:23:08.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:23:08.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:23:08.450 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:23:08.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:23:08.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:23:08.949 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:23:08.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=73d88e8f-3aaf-44a0-89c4-5dae4a1685b5 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.149458/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:08.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:23:08.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:23:08.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:23:08.967 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:08.964+0000 7f25742ad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:08.969 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:08.966+0000 7f25742ad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:08.970 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:08.967+0000 7f25742ad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:09.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:23:09.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:09.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:23:09.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:09.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:09.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:09.308 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:23:09.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:09.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:09.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:09.485 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:09.481+0000 7f25742ad0c0 -1 Falling back to public interface 2021-05-12T12:23:09.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:10.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:10.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:10.690 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:23:10.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:10.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:10.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:11.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:11.664 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:11.661+0000 7f25742ad0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:23:12.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:12.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:12.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:23:12.051 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:23:12.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:12.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:12.425 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/151963,v1:127.0.0.1:6811/151963] [v2:127.0.0.1:6812/151963,v1:127.0.0.1:6813/151963] exists,up e295ad2b-de5d-4739-940c-fa9c524ec04d 2021-05-12T12:23:12.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:12.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:12.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:12.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2021-05-12T12:23:12.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:23:12.803 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' does not exist 2021-05-12T12:23:12.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:23:12.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:23:13.379 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' already exists 2021-05-12T12:23:13.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:23:14.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:23:15.340 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2021-05-12T12:23:15.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2021-05-12T12:23:15.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2021-05-12T12:23:15.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2021-05-12T12:23:15.647 INFO:tasks.workunit.client.0.smithi191.stdout:rbd/foo2 mtime 2021-05-12T12:23:15.000000+0000, size 825480 2021-05-12T12:23:15.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2021-05-12T12:23:15.839 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0: {} 2021-05-12T12:23:15.892 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1: {} 2021-05-12T12:23:15.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2021-05-12T12:23:15.941 INFO:tasks.workunit.client.0.smithi191.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2021-05-12T12:23:15.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2021-05-12T12:23:15.976 INFO:tasks.workunit.client.0.smithi191.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2021-05-12T12:23:15.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2021-05-12T12:23:16.144 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0: {} 2021-05-12T12:23:16.175 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1: {} 2021-05-12T12:23:16.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2021-05-12T12:23:16.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2021-05-12T12:23:16.360 INFO:tasks.workunit.client.0.smithi191.stdout:rbd/foo3 mtime 2021-05-12T12:23:15.000000+0000, size 825480 2021-05-12T12:23:16.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2021-05-12T12:23:16.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-05-12T12:23:16.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:23:16.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-05-12T12:23:16.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:16.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:16.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:16.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:16.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:16.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:16.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:16.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:16.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:16.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:16.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:16.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:16.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:16.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:16.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:16.474 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:16.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:16.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:23:16.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-05-12T12:23:16.479 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:16.479 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:16.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:23:16.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.149458 2021-05-12T12:23:16.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:16.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:16.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:23:16.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-copy-from 0 2021-05-12T12:23:16.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-05-12T12:23:16.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:23:16.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-05-12T12:23:16.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:16.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:16.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:16.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:16.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:16.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:16.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:16.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:16.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:16.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:16.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:16.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:16.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:16.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:16.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:16.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:16.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:16.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:23:16.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-05-12T12:23:16.491 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:16.492 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:16.492 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.149458 2021-05-12T12:23:16.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.149458 2021-05-12T12:23:16.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:16.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:16.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:23:16.493 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:23:16.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:23:16.555 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2021-05-12T12:23:16.556 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2021-05-12T12:23:16.615 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:23:16.625 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:23:16.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:23:16.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:23:16.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:23:16.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:23:16.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:23:16.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:23:16.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:23:16.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:23:16.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-dup 2021-05-12T12:23:16.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2021-05-12T12:23:16.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2021-05-12T12:23:16.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-05-12T12:23:16.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-05-12T12:23:16.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2021-05-12T12:23:16.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2021-05-12T12:23:16.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none ' 2021-05-12T12:23:16.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-05-12T12:23:16.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2021-05-12T12:23:16.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:16.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2021-05-12T12:23:16.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:23:16.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs=TEST_filestore_to_bluestore 2021-05-12T12:23:16.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:19: run: for func in $funcs 2021-05-12T12:23:16.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:20: run: setup td/osd-dup 2021-05-12T12:23:16.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-dup 2021-05-12T12:23:16.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-dup 2021-05-12T12:23:16.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-05-12T12:23:16.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:23:16.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-05-12T12:23:16.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:16.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:16.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:16.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:16.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:16.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:16.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:16.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:16.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:16.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:16.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:16.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:16.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:16.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:16.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:16.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:16.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:16.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:23:16.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-05-12T12:23:16.641 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:16.641 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:16.641 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:16.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.152848 2021-05-12T12:23:16.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:16.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:16.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-dup 2021-05-12T12:23:16.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:23:16.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:16.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:16.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.152848 2021-05-12T12:23:16.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:23:16.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:23:16.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:23:16.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:23:16.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-dup 1' TERM HUP INT 2021-05-12T12:23:16.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:21: run: TEST_filestore_to_bluestore td/osd-dup 2021-05-12T12:23:16.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:27: TEST_filestore_to_bluestore: local dir=td/osd-dup 2021-05-12T12:23:16.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: ulimit -n 2021-05-12T12:23:16.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: local flimit=4096 2021-05-12T12:23:16.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:30: TEST_filestore_to_bluestore: '[' 4096 -lt 1536 ']' 2021-05-12T12:23:16.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:34: TEST_filestore_to_bluestore: run_mon td/osd-dup a 2021-05-12T12:23:16.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-dup 2021-05-12T12:23:16.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:23:16.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:23:16.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:23:16.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-dup/a 2021-05-12T12:23:16.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-dup/a --run-dir=td/osd-dup 2021-05-12T12:23:17.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:23:17.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:17.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:17.004 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:17.004 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:17.004 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:17.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:17.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-dup/a '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' --mon-cluster-log-file=td/osd-dup/log --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:23:17.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:23:17.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:23:17.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:23:17.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:23:17.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:23:17.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:23:17.266 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:23:17.266 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:23:17.266 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:23:17.267 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:23:17.267 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:17.267 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:17.267 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.152848/ceph-mon.a.asok 2021-05-12T12:23:17.267 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:23:17.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.152848/ceph-mon.a.asok config get fsid 2021-05-12T12:23:17.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:23:17.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:23:17.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:23:17.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:23:17.362 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:23:17.362 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:23:17.362 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:23:17.362 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:23:17.362 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:23:17.363 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:17.363 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:17.363 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.152848/ceph-mon.a.asok 2021-05-12T12:23:17.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:23:17.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.152848/ceph-mon.a.asok config get mon_host 2021-05-12T12:23:17.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:35: TEST_filestore_to_bluestore: run_mgr td/osd-dup x 2021-05-12T12:23:17.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-dup 2021-05-12T12:23:17.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:23:17.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:23:17.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:23:17.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-dup/x 2021-05-12T12:23:17.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:23:18.316 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:23:18.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:18.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:18.317 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:18.317 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:18.318 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:18.318 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:18.318 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:23:18.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-dup/x '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:23:18.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:36: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 0 2021-05-12T12:23:18.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: local dir=td/osd-dup 2021-05-12T12:23:18.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: shift 2021-05-12T12:23:18.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: local id=0 2021-05-12T12:23:18.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: shift 2021-05-12T12:23:18.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: local osd_data=td/osd-dup/0 2021-05-12T12:23:18.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: local 'ceph_args=--fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:18.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:18.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:18.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:18.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/0' 2021-05-12T12:23:18.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-05-12T12:23:18.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --chdir=' 2021-05-12T12:23:18.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+= 2021-05-12T12:23:18.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T12:23:18.346 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: get_asok_path 2021-05-12T12:23:18.346 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:18.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:18.347 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:18.347 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:18.347 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:18.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:18.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:18.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:18.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-05-12T12:23:18.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-05-12T12:23:18.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T12:23:18.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T12:23:18.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:18.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:18.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:18.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' ' 2021-05-12T12:23:18.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+= 2021-05-12T12:23:18.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: mkdir -p td/osd-dup/0 2021-05-12T12:23:18.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: uuidgen 2021-05-12T12:23:18.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: local uuid=8739d95d-5e92-471e-81a3-d1f565c5d976 2021-05-12T12:23:18.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: echo 'add osd 8739d95d-5e92-471e-81a3-d1f565c5d976' 2021-05-12T12:23:18.351 INFO:tasks.workunit.client.0.smithi191.stdout:add osd 8739d95d-5e92-471e-81a3-d1f565c5d976 2021-05-12T12:23:18.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: ceph-authtool --gen-print-key 2021-05-12T12:23:18.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: OSD_SECRET=AQC2yJtge04VFRAAyfnPq4tzB6tnSOarkFOvuA== 2021-05-12T12:23:18.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo '{"cephx_secret": "AQC2yJtge04VFRAAyfnPq4tzB6tnSOarkFOvuA=="}' 2021-05-12T12:23:18.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph osd new 8739d95d-5e92-471e-81a3-d1f565c5d976 -i td/osd-dup/0/new.json 2021-05-12T12:23:18.663 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:23:18.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: rm td/osd-dup/0/new.json 2021-05-12T12:23:18.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph-osd -i 0 --fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC2yJtge04VFRAAyfnPq4tzB6tnSOarkFOvuA== --osd-uuid 8739d95d-5e92-471e-81a3-d1f565c5d976 --osd-objectstore=filestore 2021-05-12T12:23:18.697 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:18.694+0000 7f3928c730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:18.699 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:18.696+0000 7f3928c730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:18.701 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:18.698+0000 7f3928c730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:19.089 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:19.086+0000 7f3928c730c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:19.584 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:19.581+0000 7f3928c730c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:19.585 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:19.581+0000 7f3928c730c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:19.585 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:19.581+0000 7f3928c730c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:20.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: local key_fn=td/osd-dup/0/keyring 2021-05-12T12:23:20.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: cat 2021-05-12T12:23:20.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: echo adding osd0 key to auth repository 2021-05-12T12:23:20.023 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:23:20.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: ceph -i td/osd-dup/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:23:20.193 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-05-12T12:23:20.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo start osd.0 2021-05-12T12:23:20.194 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:23:20.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph-osd -i 0 --fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:20.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: ceph osd dump --format=json 2021-05-12T12:23:20.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: jq '.flags_set[]' 2021-05-12T12:23:20.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: grep -q '"noup"' 2021-05-12T12:23:20.215 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:20.212+0000 7f074281b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:20.217 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:20.214+0000 7f074281b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:20.219 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:20.216+0000 7f074281b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:20.221 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:20.218+0000 7f074281b0c0 -1 Falling back to public interface 2021-05-12T12:23:20.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: run_osd_filestore: wait_for_osd up 0 2021-05-12T12:23:20.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:20.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:23:20.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:20.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:20.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:20.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:20.500 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:23:20.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:20.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:20.648 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:20.644+0000 7f074281b0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:20.648 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:20.645+0000 7f074281b0c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:20.648 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:20.645+0000 7f074281b0c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:20.653 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:20.650+0000 7f074281b0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:23:20.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:21.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:21.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:21.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:21.745 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:23:21.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:21.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:21.983 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/153169,v1:127.0.0.1:6801/153169] [v2:127.0.0.1:6802/153169,v1:127.0.0.1:6803/153169] exists,up 8739d95d-5e92-471e-81a3-d1f565c5d976 2021-05-12T12:23:21.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:21.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:21.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:21.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: cat td/osd-dup/osd.0.pid 2021-05-12T12:23:21.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: osd_pid=153169 2021-05-12T12:23:21.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:38: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 1 2021-05-12T12:23:21.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: local dir=td/osd-dup 2021-05-12T12:23:21.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: shift 2021-05-12T12:23:21.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: local id=1 2021-05-12T12:23:21.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: shift 2021-05-12T12:23:21.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: local osd_data=td/osd-dup/1 2021-05-12T12:23:21.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: local 'ceph_args=--fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:21.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:21.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:21.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:21.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/1' 2021-05-12T12:23:21.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/1/journal' 2021-05-12T12:23:21.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --chdir=' 2021-05-12T12:23:21.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+= 2021-05-12T12:23:21.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T12:23:21.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: get_asok_path 2021-05-12T12:23:21.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:21.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:21.993 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:21.993 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:21.993 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:21.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:21.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:21.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:21.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-05-12T12:23:21.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-05-12T12:23:21.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T12:23:21.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T12:23:21.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:21.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:21.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:21.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' ' 2021-05-12T12:23:21.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+= 2021-05-12T12:23:21.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: mkdir -p td/osd-dup/1 2021-05-12T12:23:21.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: uuidgen 2021-05-12T12:23:21.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: local uuid=f77b6f78-bc37-4ce8-b49a-09649d596f3a 2021-05-12T12:23:21.997 INFO:tasks.workunit.client.0.smithi191.stdout:add osd f77b6f78-bc37-4ce8-b49a-09649d596f3a 2021-05-12T12:23:21.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: echo 'add osd f77b6f78-bc37-4ce8-b49a-09649d596f3a' 2021-05-12T12:23:21.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: ceph-authtool --gen-print-key 2021-05-12T12:23:22.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: OSD_SECRET=AQC5yJtguCWFOxAA+H6llAlXGirX4/PmO9r6/Q== 2021-05-12T12:23:22.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo '{"cephx_secret": "AQC5yJtguCWFOxAA+H6llAlXGirX4/PmO9r6/Q=="}' 2021-05-12T12:23:22.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph osd new f77b6f78-bc37-4ce8-b49a-09649d596f3a -i td/osd-dup/1/new.json 2021-05-12T12:23:22.318 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:23:22.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: rm td/osd-dup/1/new.json 2021-05-12T12:23:22.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph-osd -i 1 --fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC5yJtguCWFOxAA+H6llAlXGirX4/PmO9r6/Q== --osd-uuid f77b6f78-bc37-4ce8-b49a-09649d596f3a --osd-objectstore=filestore 2021-05-12T12:23:22.352 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:22.349+0000 7f7ea463b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:22.360 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:22.357+0000 7f7ea463b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:22.362 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:22.359+0000 7f7ea463b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:22.729 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:22.726+0000 7f7ea463b0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:23.442 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:23.439+0000 7f7ea463b0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:23.450 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:23.447+0000 7f7ea463b0c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:23.450 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:23.447+0000 7f7ea463b0c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:23.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: local key_fn=td/osd-dup/1/keyring 2021-05-12T12:23:23.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: cat 2021-05-12T12:23:23.970 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:23:23.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: echo adding osd1 key to auth repository 2021-05-12T12:23:23.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: ceph -i td/osd-dup/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:23:24.280 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2021-05-12T12:23:24.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo start osd.1 2021-05-12T12:23:24.282 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:23:24.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph-osd -i 1 --fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:24.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: ceph osd dump --format=json 2021-05-12T12:23:24.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: jq '.flags_set[]' 2021-05-12T12:23:24.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: grep -q '"noup"' 2021-05-12T12:23:24.303 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:24.300+0000 7fabcbeec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:24.304 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:24.301+0000 7fabcbeec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:24.305 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:24.302+0000 7fabcbeec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:24.307 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:24.304+0000 7fabcbeec0c0 -1 Falling back to public interface 2021-05-12T12:23:24.689 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:24.686+0000 7fabcbeec0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:24.689 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:24.686+0000 7fabcbeec0c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:24.689 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:24.686+0000 7fabcbeec0c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:24.694 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:24.691+0000 7fabcbeec0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:23:24.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: run_osd_filestore: wait_for_osd up 1 2021-05-12T12:23:24.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:24.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:23:24.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:24.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:24.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:24.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:24.769 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:23:24.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:24.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:25.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:26.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:26.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:26.137 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:23:26.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:26.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:26.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:26.501 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/153551,v1:127.0.0.1:6811/153551] [v2:127.0.0.1:6812/153551,v1:127.0.0.1:6813/153551] exists,up f77b6f78-bc37-4ce8-b49a-09649d596f3a 2021-05-12T12:23:26.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:26.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:26.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:26.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:39: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 2 2021-05-12T12:23:26.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: local dir=td/osd-dup 2021-05-12T12:23:26.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: shift 2021-05-12T12:23:26.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: local id=2 2021-05-12T12:23:26.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: shift 2021-05-12T12:23:26.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: local osd_data=td/osd-dup/2 2021-05-12T12:23:26.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: local 'ceph_args=--fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:26.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:26.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:26.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:26.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/2' 2021-05-12T12:23:26.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/2/journal' 2021-05-12T12:23:26.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --chdir=' 2021-05-12T12:23:26.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+= 2021-05-12T12:23:26.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T12:23:26.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: get_asok_path 2021-05-12T12:23:26.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:26.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:26.510 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:26.510 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:26.510 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:26.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:26.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:26.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:26.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-05-12T12:23:26.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-05-12T12:23:26.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T12:23:26.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T12:23:26.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:26.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:26.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:26.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' ' 2021-05-12T12:23:26.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+= 2021-05-12T12:23:26.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: mkdir -p td/osd-dup/2 2021-05-12T12:23:26.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: uuidgen 2021-05-12T12:23:26.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: local uuid=377559ad-3729-47b2-b9a8-0b6c648ea91a 2021-05-12T12:23:26.514 INFO:tasks.workunit.client.0.smithi191.stdout:add osd 377559ad-3729-47b2-b9a8-0b6c648ea91a 2021-05-12T12:23:26.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: echo 'add osd 377559ad-3729-47b2-b9a8-0b6c648ea91a' 2021-05-12T12:23:26.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: ceph-authtool --gen-print-key 2021-05-12T12:23:26.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: OSD_SECRET=AQC+yJtgQtLIHhAAuvldOdqXQCJlhAUujnHc2A== 2021-05-12T12:23:26.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo '{"cephx_secret": "AQC+yJtgQtLIHhAAuvldOdqXQCJlhAUujnHc2A=="}' 2021-05-12T12:23:26.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph osd new 377559ad-3729-47b2-b9a8-0b6c648ea91a -i td/osd-dup/2/new.json 2021-05-12T12:23:26.985 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:23:26.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: rm td/osd-dup/2/new.json 2021-05-12T12:23:26.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph-osd -i 2 --fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC+yJtgQtLIHhAAuvldOdqXQCJlhAUujnHc2A== --osd-uuid 377559ad-3729-47b2-b9a8-0b6c648ea91a --osd-objectstore=filestore 2021-05-12T12:23:27.019 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:27.016+0000 7f64d2eb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:27.021 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:27.018+0000 7f64d2eb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:27.022 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:27.019+0000 7f64d2eb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:27.606 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:27.604+0000 7f64d2eb50c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:28.118 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:28.115+0000 7f64d2eb50c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:28.119 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:28.115+0000 7f64d2eb50c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:28.119 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:28.115+0000 7f64d2eb50c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:28.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: local key_fn=td/osd-dup/2/keyring 2021-05-12T12:23:28.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: cat 2021-05-12T12:23:28.564 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:23:28.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: echo adding osd2 key to auth repository 2021-05-12T12:23:28.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: ceph -i td/osd-dup/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:23:28.881 INFO:tasks.workunit.client.0.smithi191.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2021-05-12T12:23:28.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo start osd.2 2021-05-12T12:23:28.883 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:23:28.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph-osd -i 2 --fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:28.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: ceph osd dump --format=json 2021-05-12T12:23:28.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: jq '.flags_set[]' 2021-05-12T12:23:28.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: grep -q '"noup"' 2021-05-12T12:23:28.903 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:28.900+0000 7fb05c93a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:28.905 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:28.902+0000 7fb05c93a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:28.906 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:28.903+0000 7fb05c93a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:28.908 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:28.905+0000 7fb05c93a0c0 -1 Falling back to public interface 2021-05-12T12:23:29.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: run_osd_filestore: wait_for_osd up 2 2021-05-12T12:23:29.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:29.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:23:29.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:29.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:29.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:29.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:29.361 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:23:29.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:29.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:23:29.399 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:29.396+0000 7fb05c93a0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:29.400 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:29.396+0000 7fb05c93a0c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:29.400 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:29.396+0000 7fb05c93a0c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:29.405 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:29.402+0000 7fb05c93a0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:23:29.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:30.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:30.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:30.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:30.721 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:23:30.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:30.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:23:31.080 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/153851,v1:127.0.0.1:6819/153851] [v2:127.0.0.1:6820/153851,v1:127.0.0.1:6821/153851] exists,up 377559ad-3729-47b2-b9a8-0b6c648ea91a 2021-05-12T12:23:31.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:31.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:31.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:31.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:41: TEST_filestore_to_bluestore: sleep 5 2021-05-12T12:23:36.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:43: TEST_filestore_to_bluestore: create_pool foo 16 2021-05-12T12:23:36.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2021-05-12T12:23:36.805 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foo' already exists 2021-05-12T12:23:36.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:23:37.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:46: TEST_filestore_to_bluestore: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2021-05-12T12:23:37.846 INFO:tasks.workunit.client.0.smithi191.stdout:hints = 1 2021-05-12T12:23:37.846 INFO:tasks.workunit.client.0.smithi191.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2021-05-12T12:23:37.846 INFO:tasks.workunit.client.0.smithi191.stdout:Object prefix: benchmark_data_smithi191_154040 2021-05-12T12:23:37.847 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:23:37.847 INFO:tasks.workunit.client.0.smithi191.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:23:38.846 INFO:tasks.workunit.client.0.smithi191.stdout: 1 16 16 0 0 0 - 0 2021-05-12T12:23:39.846 INFO:tasks.workunit.client.0.smithi191.stdout: 2 16 16 0 0 0 - 0 2021-05-12T12:23:40.846 INFO:tasks.workunit.client.0.smithi191.stdout: 3 16 17 1 0.00130193 0.00130208 2.71751 2.71751 2021-05-12T12:23:41.846 INFO:tasks.workunit.client.0.smithi191.stdout: 4 16 18 2 0.00195289 0.00390625 3.10799 2.91275 2021-05-12T12:23:42.846 INFO:tasks.workunit.client.0.smithi191.stdout: 5 16 18 2 0.00156231 0 - 2.91275 2021-05-12T12:23:43.847 INFO:tasks.workunit.client.0.smithi191.stdout: 6 16 97 81 0.0527278 0.154297 0.121379 1.17191 2021-05-12T12:23:44.847 INFO:tasks.workunit.client.0.smithi191.stdout: 7 16 225 209 0.116614 0.5 0.122727 0.530671 2021-05-12T12:23:45.847 INFO:tasks.workunit.client.0.smithi191.stdout: 8 16 305 289 0.141095 0.3125 0.280501 0.427005 2021-05-12T12:23:46.847 INFO:tasks.workunit.client.0.smithi191.stdout: 9 16 388 372 0.161438 0.324219 0.130614 0.381131 2021-05-12T12:23:47.847 INFO:tasks.workunit.client.0.smithi191.stdout: 10 16 481 465 0.181617 0.363281 0.154513 0.341889 2021-05-12T12:23:48.847 INFO:tasks.workunit.client.0.smithi191.stdout:Total time run: 10.0621 2021-05-12T12:23:48.848 INFO:tasks.workunit.client.0.smithi191.stdout:Total writes made: 481 2021-05-12T12:23:48.848 INFO:tasks.workunit.client.0.smithi191.stdout:Write size: 4096 2021-05-12T12:23:48.848 INFO:tasks.workunit.client.0.smithi191.stdout:Object size: 4096 2021-05-12T12:23:48.848 INFO:tasks.workunit.client.0.smithi191.stdout:Bandwidth (MB/sec): 0.18673 2021-05-12T12:23:48.849 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Bandwidth: 0.192405 2021-05-12T12:23:48.849 INFO:tasks.workunit.client.0.smithi191.stdout:Max bandwidth (MB/sec): 0.5 2021-05-12T12:23:48.849 INFO:tasks.workunit.client.0.smithi191.stdout:Min bandwidth (MB/sec): 0 2021-05-12T12:23:48.849 INFO:tasks.workunit.client.0.smithi191.stdout:Average IOPS: 47 2021-05-12T12:23:48.850 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev IOPS: 49.291 2021-05-12T12:23:48.850 INFO:tasks.workunit.client.0.smithi191.stdout:Max IOPS: 128 2021-05-12T12:23:48.850 INFO:tasks.workunit.client.0.smithi191.stdout:Min IOPS: 0 2021-05-12T12:23:48.850 INFO:tasks.workunit.client.0.smithi191.stdout:Average Latency(s): 0.334356 2021-05-12T12:23:48.850 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Latency(s): 0.908339 2021-05-12T12:23:48.851 INFO:tasks.workunit.client.0.smithi191.stdout:Max latency(s): 5.40515 2021-05-12T12:23:48.851 INFO:tasks.workunit.client.0.smithi191.stdout:Min latency(s): 0.113515 2021-05-12T12:23:48.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 153169 2021-05-12T12:23:48.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: sleep 1 2021-05-12T12:23:49.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 153169 2021-05-12T12:23:49.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh: line 49: kill: (153169) - No such process 2021-05-12T12:23:49.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:50: TEST_filestore_to_bluestore: ceph osd down 0 2021-05-12T12:23:50.596 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already down. 2021-05-12T12:23:50.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:52: TEST_filestore_to_bluestore: mv td/osd-dup/0 td/osd-dup/0.old 2021-05-12T12:23:50.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:53: TEST_filestore_to_bluestore: mkdir td/osd-dup/0 2021-05-12T12:23:50.611 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: cat td/osd-dup/0.old/fsid 2021-05-12T12:23:50.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: ofsid=8739d95d-5e92-471e-81a3-d1f565c5d976 2021-05-12T12:23:50.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:55: TEST_filestore_to_bluestore: echo 'osd fsid 8739d95d-5e92-471e-81a3-d1f565c5d976' 2021-05-12T12:23:50.613 INFO:tasks.workunit.client.0.smithi191.stdout:osd fsid 8739d95d-5e92-471e-81a3-d1f565c5d976 2021-05-12T12:23:50.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:56: TEST_filestore_to_bluestore: O='--fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:50.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:57: TEST_filestore_to_bluestore: CEPH_ARGS+='--log-file td/osd-dup/cot.log --log-max-recent 0 ' 2021-05-12T12:23:50.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:58: TEST_filestore_to_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-dup/0 --fsid 8739d95d-5e92-471e-81a3-d1f565c5d976 --op mkfs --no-mon-config 2021-05-12T12:23:54.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:60: TEST_filestore_to_bluestore: ceph-objectstore-tool --data-path td/osd-dup/0.old --target-data-path td/osd-dup/0 --op dup 2021-05-12T12:23:54.341 INFO:tasks.workunit.client.0.smithi191.stdout:dup from filestore: td/osd-dup/0.old 2021-05-12T12:23:54.341 INFO:tasks.workunit.client.0.smithi191.stdout: to bluestore: td/osd-dup/0 2021-05-12T12:23:55.301 INFO:tasks.workunit.client.0.smithi191.stdout:fsid 8739d95d-5e92-471e-81a3-d1f565c5d976 2021-05-12T12:23:55.302 INFO:tasks.workunit.client.0.smithi191.stdout:17 collections 2021-05-12T12:23:55.302 INFO:tasks.workunit.client.0.smithi191.stdout:1/17 1.b_head 2021-05-12T12:23:55.786 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 40 objects, 159744 bytes, 45 keys 2021-05-12T12:23:55.786 INFO:tasks.workunit.client.0.smithi191.stdout:2/17 1.7_head 2021-05-12T12:23:55.954 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 24 objects, 94208 bytes, 29 keys 2021-05-12T12:23:55.955 INFO:tasks.workunit.client.0.smithi191.stdout:3/17 1.4_head 2021-05-12T12:23:56.167 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2021-05-12T12:23:56.167 INFO:tasks.workunit.client.0.smithi191.stdout:4/17 1.a_head 2021-05-12T12:23:56.360 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 37 objects, 147456 bytes, 42 keys 2021-05-12T12:23:56.360 INFO:tasks.workunit.client.0.smithi191.stdout:5/17 1.d_head 2021-05-12T12:23:56.654 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 44 objects, 172056 bytes, 49 keys 2021-05-12T12:23:56.654 INFO:tasks.workunit.client.0.smithi191.stdout:6/17 1.8_head 2021-05-12T12:23:56.816 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 25 objects, 98304 bytes, 30 keys 2021-05-12T12:23:56.816 INFO:tasks.workunit.client.0.smithi191.stdout:7/17 1.0_head 2021-05-12T12:23:56.933 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2021-05-12T12:23:56.933 INFO:tasks.workunit.client.0.smithi191.stdout:8/17 1.5_head 2021-05-12T12:23:57.151 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 31 objects, 122880 bytes, 36 keys 2021-05-12T12:23:57.151 INFO:tasks.workunit.client.0.smithi191.stdout:9/17 1.3_head 2021-05-12T12:23:57.345 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 31 objects, 122880 bytes, 36 keys 2021-05-12T12:23:57.346 INFO:tasks.workunit.client.0.smithi191.stdout:10/17 1.6_head 2021-05-12T12:23:57.572 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 35 objects, 139264 bytes, 40 keys 2021-05-12T12:23:57.572 INFO:tasks.workunit.client.0.smithi191.stdout:11/17 1.1_head 2021-05-12T12:23:57.673 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 24 objects, 94208 bytes, 29 keys 2021-05-12T12:23:57.674 INFO:tasks.workunit.client.0.smithi191.stdout:12/17 1.9_head 2021-05-12T12:23:57.809 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 29 objects, 114688 bytes, 34 keys 2021-05-12T12:23:57.809 INFO:tasks.workunit.client.0.smithi191.stdout:13/17 1.e_head 2021-05-12T12:23:58.019 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 30 objects, 118784 bytes, 35 keys 2021-05-12T12:23:58.020 INFO:tasks.workunit.client.0.smithi191.stdout:14/17 meta 2021-05-12T12:23:58.023 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 46 objects, 47128 bytes, 0 keys 2021-05-12T12:23:58.023 INFO:tasks.workunit.client.0.smithi191.stdout:15/17 1.c_head 2021-05-12T12:23:58.179 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 32 objects, 126976 bytes, 37 keys 2021-05-12T12:23:58.180 INFO:tasks.workunit.client.0.smithi191.stdout:16/17 1.2_head 2021-05-12T12:23:58.182 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 33 objects, 131072 bytes, 38 keys 2021-05-12T12:23:58.182 INFO:tasks.workunit.client.0.smithi191.stdout:17/17 1.f_head 2021-05-12T12:23:58.365 INFO:tasks.workunit.client.0.smithi191.stdout: 0 objects, 0 bytes, 0 keys 27 objects, 106496 bytes, 32 keys 2021-05-12T12:23:58.366 INFO:tasks.workunit.client.0.smithi191.stdout:keyring 2021-05-12T12:23:58.366 INFO:tasks.workunit.client.0.smithi191.stdout:duping osd metadata 2021-05-12T12:23:58.726 INFO:tasks.workunit.client.0.smithi191.stdout:done. 2021-05-12T12:23:59.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:62: TEST_filestore_to_bluestore: CEPH_ARGS='--fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:59.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:64: TEST_filestore_to_bluestore: activate_osd td/osd-dup 0 2021-05-12T12:23:59.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-dup 2021-05-12T12:23:59.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:23:59.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:23:59.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:23:59.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-dup/0 2021-05-12T12:23:59.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:59.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:59.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:59.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:59.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-dup/0' 2021-05-12T12:23:59.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-05-12T12:23:59.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:23:59.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:23:59.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T12:23:59.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:23:59.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:59.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:59.513 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:59.514 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:59.514 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:23:59.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:59.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' 2021-05-12T12:23:59.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:59.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T12:23:59.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T12:23:59.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:59.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:59.516 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:59.516 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:23:59.516 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:23:59.516 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-dup/0 2021-05-12T12:23:59.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:23:59.517 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:23:59.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=c4d126a6-2deb-4563-adec-785b38cca76f --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.152848/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:59.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-dup/0/whoami 2021-05-12T12:23:59.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:23:59.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:23:59.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:23:59.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:23:59.533 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:59.530+0000 7f8ed8c1c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:59.535 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:59.532+0000 7f8ed8c1c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:59.537 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:23:59.534+0000 7f8ed8c1c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:59.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:23:59.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:59.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:23:59.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:59.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:59.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:59.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:59.880 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:23:59.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:59.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:00.054 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:00.050+0000 7f8ed8c1c0c0 -1 Falling back to public interface 2021-05-12T12:24:00.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:01.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:01.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:01.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:24:01.237 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:24:01.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:01.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:01.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:02.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:02.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:02.609 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:24:02.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:24:02.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:02.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:02.660 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:02.656+0000 7f8ed8c1c0c0 -1 osd.0 21 log_to_monitors {default=true} 2021-05-12T12:24:03.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:04.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:04.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:04.005 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:24:04.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:24:04.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:04.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:04.370 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/154913,v1:127.0.0.1:6801/154913] [v2:127.0.0.1:6802/154913,v1:127.0.0.1:6803/154913] exists,up 8739d95d-5e92-471e-81a3-d1f565c5d976 2021-05-12T12:24:04.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:24:04.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:24:04.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:24:04.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: ceph osd stat 2021-05-12T12:24:04.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: grep '3 up' 2021-05-12T12:24:04.733 INFO:tasks.workunit.client.0.smithi191.stdout:3 osds: 3 up (since 1.76152s), 3 in (since 37s); epoch: e28 2021-05-12T12:24:04.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: ceph osd metadata 0 2021-05-12T12:24:04.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: grep bluestore 2021-05-12T12:24:05.088 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:24:05.088 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:24:05.089 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:24:05.089 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:24:05.089 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_path": "td/osd-dup/0/block", 2021-05-12T12:24:05.089 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:24:05.090 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:24:05.090 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:24:05.090 INFO:tasks.workunit.client.0.smithi191.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:24:05.090 INFO:tasks.workunit.client.0.smithi191.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:24:05.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:69: TEST_filestore_to_bluestore: ceph osd scrub 0 2021-05-12T12:24:05.435 INFO:tasks.workunit.client.0.smithi191.stderr:instructed osd(s) 0 to scrub 2021-05-12T12:24:05.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:72: TEST_filestore_to_bluestore: sleep 15 2021-05-12T12:24:20.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:74: TEST_filestore_to_bluestore: flush_pg_stats 2021-05-12T12:24:20.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:24:20.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:24:20.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:24:20.812 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:24:20.812 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:24:20.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:24:20.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:20.813 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:24:20.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=111669149702 2021-05-12T12:24:20.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 111669149702 2021-05-12T12:24:20.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149702' 2021-05-12T12:24:20.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:20.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:24:21.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672974 2021-05-12T12:24:21.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672974 2021-05-12T12:24:21.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149702 1-42949672974' 2021-05-12T12:24:21.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:21.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:24:21.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509453 2021-05-12T12:24:21.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509453 2021-05-12T12:24:21.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149702 1-42949672974 2-64424509453' 2021-05-12T12:24:21.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:21.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-111669149702 2021-05-12T12:24:21.190 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:21.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:24:21.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-111669149702 2021-05-12T12:24:21.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:21.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=111669149702 2021-05-12T12:24:21.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2021-05-12T12:24:21.192 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 111669149702 2021-05-12T12:24:21.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:24:21.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 111669149702 -lt 111669149702 2021-05-12T12:24:21.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:21.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-05-12T12:24:21.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:21.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:24:21.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672974 2021-05-12T12:24:21.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:21.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672974 2021-05-12T12:24:21.555 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672974 2021-05-12T12:24:21.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-05-12T12:24:21.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:24:21.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672974 -lt 42949672974 2021-05-12T12:24:21.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:21.912 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509453 2021-05-12T12:24:21.912 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:21.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:24:21.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509453 2021-05-12T12:24:21.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:21.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509453 2021-05-12T12:24:21.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-05-12T12:24:21.915 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 64424509453 2021-05-12T12:24:21.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:24:22.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509451 -lt 64424509453 2021-05-12T12:24:22.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:24:23.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:24:23.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:24:23.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509453 -lt 64424509453 2021-05-12T12:24:23.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:76: TEST_filestore_to_bluestore: wait_for_clean 2021-05-12T12:24:23.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:24:23.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:24:23.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:24:23.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:24:23.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:24:23.647 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:24:23.647 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:24:23.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:24:23.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:24:23.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:24:23.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:24:23.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:24:23.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:24:23.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:24:23.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:24:24.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:24:24.092 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:24:24.092 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:24:24.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:24:24.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:24.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:24:24.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=111669149705 2021-05-12T12:24:24.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 111669149705 2021-05-12T12:24:24.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149705' 2021-05-12T12:24:24.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:24.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:24:24.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672976 2021-05-12T12:24:24.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672976 2021-05-12T12:24:24.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149705 1-42949672976' 2021-05-12T12:24:24.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:24.378 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:24:24.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509455 2021-05-12T12:24:24.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509455 2021-05-12T12:24:24.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149705 1-42949672976 2-64424509455' 2021-05-12T12:24:24.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:24.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-111669149705 2021-05-12T12:24:24.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:24.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:24:24.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-111669149705 2021-05-12T12:24:24.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:24.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=111669149705 2021-05-12T12:24:24.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 111669149705' 2021-05-12T12:24:24.505 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 111669149705 2021-05-12T12:24:24.505 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:24:24.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 111669149703 -lt 111669149705 2021-05-12T12:24:24.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:24:25.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:24:25.860 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:24:26.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 111669149705 -lt 111669149705 2021-05-12T12:24:26.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:26.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672976 2021-05-12T12:24:26.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:26.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:24:26.228 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672976 2021-05-12T12:24:26.229 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:26.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672976 2021-05-12T12:24:26.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-05-12T12:24:26.230 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672976 2021-05-12T12:24:26.230 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:24:26.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672976 -lt 42949672976 2021-05-12T12:24:26.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:26.583 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509455 2021-05-12T12:24:26.583 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:26.584 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:24:26.585 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509455 2021-05-12T12:24:26.585 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:26.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509455 2021-05-12T12:24:26.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2021-05-12T12:24:26.586 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 64424509455 2021-05-12T12:24:26.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:24:26.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509455 -lt 64424509455 2021-05-12T12:24:26.943 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:24:26.943 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:24:26.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:24:27.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 16 == 0 2021-05-12T12:24:27.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:24:27.416 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:24:27.416 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:24:27.416 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:24:27.416 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:24:27.417 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:24:27.417 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:24:27.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=16 2021-05-12T12:24:27.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:24:27.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:24:27.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:24:28.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 16 = 16 2021-05-12T12:24:28.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:24:28.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:24:28.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:22: run: teardown td/osd-dup 2021-05-12T12:24:28.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-05-12T12:24:28.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:24:28.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-05-12T12:24:28.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:24:28.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:24:28.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:24:28.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:24:28.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:24:28.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:24:28.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:24:28.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:24:28.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:24:28.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:24:28.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:24:28.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:24:28.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:28.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:24:28.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:24:28.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:28.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:24:28.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:24:28.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-05-12T12:24:28.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:24:28.409 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:28.409 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:24:28.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.152848 2021-05-12T12:24:28.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:24:28.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:24:28.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:24:28.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-dup 0 2021-05-12T12:24:28.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-05-12T12:24:28.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:24:28.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-05-12T12:24:28.412 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:24:28.412 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:24:28.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:24:28.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:24:28.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:24:28.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:24:28.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:24:28.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:24:28.415 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:24:28.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:24:28.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:24:28.416 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:24:28.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:28.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:24:28.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:24:28.417 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:28.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:24:28.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:24:28.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-05-12T12:24:28.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:24:28.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:28.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.152848 2021-05-12T12:24:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.152848 2021-05-12T12:24:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:24:28.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:24:28.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:24:28.422 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:24:28.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:24:28.485 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2021-05-12T12:24:28.486 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2021-05-12T12:24:28.546 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:24:28.553 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:24:28.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:24:28.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:24:28.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:24:28.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:24:28.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:24:28.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:24:28.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:24:28.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:24:28.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-fast-mark-down 2021-05-12T12:24:28.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2021-05-12T12:24:28.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2021-05-12T12:24:28.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2021-05-12T12:24:28.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2021-05-12T12:24:28.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2021-05-12T12:24:28.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2021-05-12T12:24:28.557 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2021-05-12T12:24:28.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none ' 2021-05-12T12:24:28.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none ' 2021-05-12T12:24:28.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2021-05-12T12:24:28.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2021-05-12T12:24:28.559 INFO:tasks.workunit.client.0.smithi191.stdout:Ensuring old behavior is there... 2021-05-12T12:24:28.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2021-05-12T12:24:28.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-05-12T12:24:28.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-05-12T12:24:28.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-05-12T12:24:28.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T12:24:28.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:24:28.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T12:24:28.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:24:28.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:24:28.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:24:28.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:24:28.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:24:28.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:24:28.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:24:28.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:24:28.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:24:28.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:24:28.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:24:28.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:24:28.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:28.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:24:28.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:24:28.567 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:28.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:24:28.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:24:28.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T12:24:28.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:24:28.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:28.569 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:28.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.156252 2021-05-12T12:24:28.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:24:28.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:24:28.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-05-12T12:24:28.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:24:28.572 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:28.572 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:28.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.156252 2021-05-12T12:24:28.572 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:24:28.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:24:28.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:24:28.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:24:28.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-05-12T12:24:28.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-05-12T12:24:28.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-05-12T12:24:28.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:24:28.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:24:28.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:24:28.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-05-12T12:24:28.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-05-12T12:24:28.584 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:28.581+0000 7f4be0dd9700 -1 failed for service _ceph-mon._tcp 2021-05-12T12:24:28.584 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:28.581+0000 7f4be0dd9700 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-05-12T12:24:28.585 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T12:24:28.585 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:28.581+0000 7f4be0dd9700 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2021-05-12T12:24:28.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:24:28.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:28.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:28.906 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:28.906 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:28.906 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:28.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156252/$cluster-$name.asok' 2021-05-12T12:24:28.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2021-05-12T12:24:29.163 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:29.159+0000 7ff0c30af700 -1 failed for service _ceph-mon._tcp 2021-05-12T12:24:29.163 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:29.159+0000 7ff0c30af700 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-05-12T12:24:29.163 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T12:24:29.163 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:29.159+0000 7ff0c30af700 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2021-05-12T12:24:29.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2021-05-12T12:24:29.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2021-05-12T12:24:29.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-05-12T12:24:29.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-05-12T12:24:29.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T12:24:29.167 INFO:tasks.workunit.client.0.smithi191.stdout:Testing async msgr... 2021-05-12T12:24:29.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2021-05-12T12:24:29.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2021-05-12T12:24:29.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-05-12T12:24:29.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-05-12T12:24:29.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-05-12T12:24:29.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T12:24:29.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:24:29.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T12:24:29.171 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:24:29.172 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:24:29.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:24:29.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:24:29.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:24:29.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:24:29.173 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:24:29.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:24:29.173 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:24:29.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:24:29.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:24:29.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:24:29.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:29.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:24:29.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:24:29.176 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:29.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:24:29.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:24:29.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T12:24:29.177 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:24:29.177 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:29.177 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:29.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.156252 2021-05-12T12:24:29.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:24:29.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:24:29.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-05-12T12:24:29.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:24:29.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:29.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:29.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.156252 2021-05-12T12:24:29.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:24:29.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:24:29.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:24:29.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-05-12T12:24:29.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-05-12T12:24:29.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-05-12T12:24:29.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:24:29.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:24:29.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:24:29.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-05-12T12:24:29.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-05-12T12:24:29.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:24:29.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:29.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:29.545 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:29.545 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:29.545 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:29.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156252/$cluster-$name.asok' 2021-05-12T12:24:29.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2021-05-12T12:24:29.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:24:29.773 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:24:29.773 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:24:29.773 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:24:29.774 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:24:29.774 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:24:29.774 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:24:29.774 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:24:29.774 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:24:29.775 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:24:29.775 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:29.775 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:29.775 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.156252/ceph-mon.a.asok 2021-05-12T12:24:29.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:24:29.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.156252/ceph-mon.a.asok config get fsid 2021-05-12T12:24:29.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:24:29.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:24:29.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:24:29.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:24:29.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:24:29.869 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:24:29.869 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:24:29.870 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:24:29.870 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:24:29.870 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:29.870 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:29.871 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.156252/ceph-mon.a.asok 2021-05-12T12:24:29.871 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:24:29.871 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.156252/ceph-mon.a.asok config get mon_host 2021-05-12T12:24:29.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2021-05-12T12:24:29.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2021-05-12T12:24:29.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:24:29.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:24:29.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:24:29.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2021-05-12T12:24:29.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:24:30.842 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:24:30.842 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:30.842 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:30.843 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:30.843 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:30.843 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:30.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156252/$cluster-$name.asok' 2021-05-12T12:24:30.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:24:30.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:24:30.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-05-12T12:24:30.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2021-05-12T12:24:30.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-05-12T12:24:30.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:24:30.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:24:30.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:24:30.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2021-05-12T12:24:30.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T12:24:30.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:24:30.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:24:30.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:24:30.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2021-05-12T12:24:30.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2021-05-12T12:24:30.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:24:30.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:24:30.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-05-12T12:24:30.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:24:30.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:30.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:30.873 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:30.874 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:30.874 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:30.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156252/$cluster-$name.asok' 2021-05-12T12:24:30.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' 2021-05-12T12:24:30.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:24:30.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:24:30.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:24:30.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-05-12T12:24:30.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-05-12T12:24:30.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:24:30.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:24:30.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:30.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:24:30.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:24:30.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-fast-mark-down/0 2021-05-12T12:24:30.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:24:30.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=67bb7156-93ca-4111-aa8e-04809c267720 2021-05-12T12:24:30.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 67bb7156-93ca-4111-aa8e-04809c267720' 2021-05-12T12:24:30.878 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 67bb7156-93ca-4111-aa8e-04809c267720 2021-05-12T12:24:30.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:24:30.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD+yJtgBd6HNBAAPJtGkbSR5E+mlFtXE3cw8g== 2021-05-12T12:24:30.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD+yJtgBd6HNBAAPJtGkbSR5E+mlFtXE3cw8g=="}' 2021-05-12T12:24:30.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 67bb7156-93ca-4111-aa8e-04809c267720 -i td/osd-fast-mark-down/0/new.json 2021-05-12T12:24:31.214 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:24:31.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-fast-mark-down/0/new.json 2021-05-12T12:24:31.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD+yJtgBd6HNBAAPJtGkbSR5E+mlFtXE3cw8g== --osd-uuid 67bb7156-93ca-4111-aa8e-04809c267720 2021-05-12T12:24:31.250 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:31.247+0000 7fe94766b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:31.252 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:31.249+0000 7fe94766b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:31.253 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:31.250+0000 7fe94766b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:31.253 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:31.250+0000 7fe94766b0c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-05-12T12:24:31.253 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:31.250+0000 7fe94766b0c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-05-12T12:24:31.254 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:31.250+0000 7fe94766b0c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-05-12T12:24:31.296 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:31.293+0000 7fe94766b0c0 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2021-05-12T12:24:35.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2021-05-12T12:24:35.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:24:35.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:24:35.917 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:24:35.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:24:36.399 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:24:36.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:24:36.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:24:36.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:36.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:24:36.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:24:36.421 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:36.418+0000 7fa41736c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:36.423 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:36.420+0000 7fa41736c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:36.424 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:36.421+0000 7fa41736c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:36.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:24:36.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:24:36.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:24:36.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:24:36.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:24:36.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:36.762 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:24:36.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:24:36.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:36.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:36.940 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:36.937+0000 7fa41736c0c0 -1 Falling back to public interface 2021-05-12T12:24:37.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:38.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:38.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:38.138 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:24:38.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:24:38.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:38.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:38.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:39.130 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:39.126+0000 7fa41736c0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:24:39.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:39.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:39.511 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:24:39.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:24:39.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:39.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:39.894 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/157437,v1:127.0.0.1:6803/157437] [v2:127.0.0.1:6804/157437,v1:127.0.0.1:6805/157437] exists,up 67bb7156-93ca-4111-aa8e-04809c267720 2021-05-12T12:24:39.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:24:39.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:24:39.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:24:39.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2021-05-12T12:24:39.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=157437 2021-05-12T12:24:39.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-05-12T12:24:39.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2021-05-12T12:24:39.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-05-12T12:24:39.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:24:39.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:24:39.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:24:39.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2021-05-12T12:24:39.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T12:24:39.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:24:39.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:24:39.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:24:39.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2021-05-12T12:24:39.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2021-05-12T12:24:39.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:24:39.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:24:39.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-05-12T12:24:39.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:24:39.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:39.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:39.914 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:39.914 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:39.915 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:39.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156252/$cluster-$name.asok' 2021-05-12T12:24:39.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' 2021-05-12T12:24:39.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:24:39.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:24:39.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:24:39.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-05-12T12:24:39.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-05-12T12:24:39.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:24:39.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:24:39.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:39.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:24:39.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:24:39.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-fast-mark-down/1 2021-05-12T12:24:39.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:24:39.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=1fd687e3-3b21-4464-8356-11a7fcce80bf 2021-05-12T12:24:39.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 1fd687e3-3b21-4464-8356-11a7fcce80bf' 2021-05-12T12:24:39.922 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 1fd687e3-3b21-4464-8356-11a7fcce80bf 2021-05-12T12:24:39.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:24:39.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAHyZtgqNc5NhAAKfSf0Xa9Q6WeK/Y49WU/qg== 2021-05-12T12:24:39.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAHyZtgqNc5NhAAKfSf0Xa9Q6WeK/Y49WU/qg=="}' 2021-05-12T12:24:39.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 1fd687e3-3b21-4464-8356-11a7fcce80bf -i td/osd-fast-mark-down/1/new.json 2021-05-12T12:24:40.476 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:24:40.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-fast-mark-down/1/new.json 2021-05-12T12:24:40.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAHyZtgqNc5NhAAKfSf0Xa9Q6WeK/Y49WU/qg== --osd-uuid 1fd687e3-3b21-4464-8356-11a7fcce80bf 2021-05-12T12:24:40.512 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:40.509+0000 7f3f0b9dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:40.514 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:40.511+0000 7f3f0b9dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:40.516 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:40.512+0000 7f3f0b9dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:40.517 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:40.513+0000 7f3f0b9dd0c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-05-12T12:24:40.517 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:40.513+0000 7f3f0b9dd0c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-05-12T12:24:40.517 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:40.513+0000 7f3f0b9dd0c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-05-12T12:24:40.593 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:40.590+0000 7f3f0b9dd0c0 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2021-05-12T12:24:45.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2021-05-12T12:24:45.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:24:45.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:24:45.123 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:24:45.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:24:45.623 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:24:45.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:24:45.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:45.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:24:45.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:24:45.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:24:45.645 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:45.642+0000 7f62301680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:45.647 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:45.644+0000 7f62301680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:45.648 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:45.645+0000 7f62301680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:45.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:24:45.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:24:45.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:24:45.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:24:45.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:24:45.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:45.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:24:45.987 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:24:45.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:45.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:24:46.166 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:46.163+0000 7f62301680c0 -1 Falling back to public interface 2021-05-12T12:24:46.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:47.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:47.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:47.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:24:47.373 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:24:47.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:47.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:24:47.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:48.591 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:48.588+0000 7f62301680c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:24:48.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:48.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:48.780 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:24:48.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:24:48.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:48.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:24:49.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:50.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:50.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:50.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:24:50.165 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:24:50.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:50.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:24:50.523 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/158823,v1:127.0.0.1:6811/158823] [v2:127.0.0.1:6812/158823,v1:127.0.0.1:6813/158823] exists,up 1fd687e3-3b21-4464-8356-11a7fcce80bf 2021-05-12T12:24:50.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:24:50.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:24:50.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:24:50.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2021-05-12T12:24:50.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=158823 2021-05-12T12:24:50.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-05-12T12:24:50.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2021-05-12T12:24:50.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-05-12T12:24:50.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:24:50.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:24:50.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:24:50.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2021-05-12T12:24:50.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T12:24:50.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:24:50.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:24:50.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:24:50.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2021-05-12T12:24:50.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2021-05-12T12:24:50.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:24:50.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:24:50.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-05-12T12:24:50.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:24:50.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:50.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:50.533 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:50.533 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:50.534 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:24:50.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156252/$cluster-$name.asok' 2021-05-12T12:24:50.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' 2021-05-12T12:24:50.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:24:50.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:24:50.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:24:50.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-05-12T12:24:50.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-05-12T12:24:50.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:24:50.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:24:50.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:50.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:24:50.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:24:50.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-fast-mark-down/2 2021-05-12T12:24:50.537 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:24:50.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7541eb38-8743-4250-8cd1-e351593a7e4d 2021-05-12T12:24:50.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 7541eb38-8743-4250-8cd1-e351593a7e4d' 2021-05-12T12:24:50.537 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 7541eb38-8743-4250-8cd1-e351593a7e4d 2021-05-12T12:24:50.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:24:50.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQASyZtgq8RFIBAAi+4GG0y8Kylfs5Th344vig== 2021-05-12T12:24:50.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQASyZtgq8RFIBAAi+4GG0y8Kylfs5Th344vig=="}' 2021-05-12T12:24:50.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7541eb38-8743-4250-8cd1-e351593a7e4d -i td/osd-fast-mark-down/2/new.json 2021-05-12T12:24:51.002 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:24:51.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-fast-mark-down/2/new.json 2021-05-12T12:24:51.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQASyZtgq8RFIBAAi+4GG0y8Kylfs5Th344vig== --osd-uuid 7541eb38-8743-4250-8cd1-e351593a7e4d 2021-05-12T12:24:51.037 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:51.034+0000 7fa6629480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:51.039 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:51.036+0000 7fa6629480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:51.040 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:51.037+0000 7fa6629480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:51.041 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:51.038+0000 7fa6629480c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-05-12T12:24:51.041 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:51.038+0000 7fa6629480c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-05-12T12:24:51.041 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:51.038+0000 7fa6629480c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-05-12T12:24:51.080 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:51.077+0000 7fa6629480c0 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2021-05-12T12:24:55.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2021-05-12T12:24:55.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:24:55.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:24:55.632 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:24:55.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:24:56.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:24:56.120 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:24:56.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=9561638a-312f-458b-b4b2-4a9b638005c2 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.156252/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:56.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:24:56.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:24:56.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:24:56.142 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:56.139+0000 7fbb0e2db0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:56.144 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:56.141+0000 7fbb0e2db0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:56.145 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:56.142+0000 7fbb0e2db0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:56.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:24:56.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:24:56.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:24:56.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:24:56.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:24:56.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:56.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:24:56.484 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:24:56.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:56.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:24:56.661 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:56.658+0000 7fbb0e2db0c0 -1 Falling back to public interface 2021-05-12T12:24:56.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:57.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:57.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:57.843 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:24:57.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:24:57.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:57.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:24:58.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:58.834 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:24:58.831+0000 7fbb0e2db0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:24:59.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:59.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:59.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:24:59.221 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:24:59.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:59.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:24:59.590 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/160235,v1:127.0.0.1:6819/160235] [v2:127.0.0.1:6820/160235,v1:127.0.0.1:6821/160235] exists,up 7541eb38-8743-4250-8cd1-e351593a7e4d 2021-05-12T12:24:59.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:24:59.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:24:59.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:24:59.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2021-05-12T12:24:59.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=160235 2021-05-12T12:24:59.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2021-05-12T12:24:59.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:24:59.974 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' does not exist 2021-05-12T12:24:59.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:24:59.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:25:00.493 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' already exists 2021-05-12T12:25:00.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:25:01.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:25:02.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2021-05-12T12:25:02.913 INFO:tasks.workunit.client.0.smithi191.stdout:hints = 1 2021-05-12T12:25:02.914 INFO:tasks.workunit.client.0.smithi191.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2021-05-12T12:25:02.914 INFO:tasks.workunit.client.0.smithi191.stdout:Object prefix: benchmark_data_smithi191_160845 2021-05-12T12:25:02.914 INFO:tasks.workunit.client.0.smithi191.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:25:02.914 INFO:tasks.workunit.client.0.smithi191.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:25:03.914 INFO:tasks.workunit.client.0.smithi191.stdout: 1 16 80 64 0.249977 0.25 0.288424 0.2235 2021-05-12T12:25:04.914 INFO:tasks.workunit.client.0.smithi191.stdout:Total time run: 1.90136 2021-05-12T12:25:04.914 INFO:tasks.workunit.client.0.smithi191.stdout:Total writes made: 128 2021-05-12T12:25:04.914 INFO:tasks.workunit.client.0.smithi191.stdout:Write size: 4096 2021-05-12T12:25:04.914 INFO:tasks.workunit.client.0.smithi191.stdout:Object size: 4096 2021-05-12T12:25:04.915 INFO:tasks.workunit.client.0.smithi191.stdout:Bandwidth (MB/sec): 0.26297 2021-05-12T12:25:04.915 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Bandwidth: 0 2021-05-12T12:25:04.915 INFO:tasks.workunit.client.0.smithi191.stdout:Max bandwidth (MB/sec): 0.25 2021-05-12T12:25:04.915 INFO:tasks.workunit.client.0.smithi191.stdout:Min bandwidth (MB/sec): 0.25 2021-05-12T12:25:04.916 INFO:tasks.workunit.client.0.smithi191.stdout:Average IOPS: 67 2021-05-12T12:25:04.916 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev IOPS: 0 2021-05-12T12:25:04.916 INFO:tasks.workunit.client.0.smithi191.stdout:Max IOPS: 64 2021-05-12T12:25:04.916 INFO:tasks.workunit.client.0.smithi191.stdout:Min IOPS: 64 2021-05-12T12:25:04.916 INFO:tasks.workunit.client.0.smithi191.stdout:Average Latency(s): 0.236617 2021-05-12T12:25:04.917 INFO:tasks.workunit.client.0.smithi191.stdout:Stddev Latency(s): 0.0339851 2021-05-12T12:25:04.917 INFO:tasks.workunit.client.0.smithi191.stdout:Max latency(s): 0.288424 2021-05-12T12:25:04.917 INFO:tasks.workunit.client.0.smithi191.stdout:Min latency(s): 0.133827 2021-05-12T12:25:05.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2021-05-12T12:25:06.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2021-05-12T12:25:06.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2021-05-12T12:25:06.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-05-12T12:25:06.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2021-05-12T12:25:06.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=157437 2021-05-12T12:25:06.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 157437 -eq 0 ']' 2021-05-12T12:25:06.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=157437 2021-05-12T12:25:06.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 157437 2021-05-12T12:25:06.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-05-12T12:25:06.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-05-12T12:25:06.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-05-12T12:25:06.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-05-12T12:25:07.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-05-12T12:25:07.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-05-12T12:25:07.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-05-12T12:25:07.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-05-12T12:25:07.028 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-05-12T12:25:07.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2021-05-12T12:25:07.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2021-05-12T12:25:07.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2021-05-12T12:25:07.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-05-12T12:25:07.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2021-05-12T12:25:07.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-05-12T12:25:07.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 157437 -eq 157437 ']' 2021-05-12T12:25:07.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=160235 2021-05-12T12:25:07.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 160235 -eq 157437 ']' 2021-05-12T12:25:07.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=160235 2021-05-12T12:25:07.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 160235 2021-05-12T12:25:07.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-05-12T12:25:07.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-05-12T12:25:07.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-05-12T12:25:07.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-05-12T12:25:08.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-05-12T12:25:08.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-05-12T12:25:08.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-05-12T12:25:08.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-05-12T12:25:08.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-05-12T12:25:08.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2021-05-12T12:25:08.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2021-05-12T12:25:08.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2021-05-12T12:25:08.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-05-12T12:25:08.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2021-05-12T12:25:08.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2021-05-12T12:25:08.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2021-05-12T12:25:08.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T12:25:08.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:25:08.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T12:25:08.788 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:08.788 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:08.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:08.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:08.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:08.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:08.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:25:08.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:25:08.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:25:08.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:25:08.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:25:08.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:25:08.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:08.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:25:08.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:25:08.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:08.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:25:08.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:25:08.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T12:25:08.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:25:08.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:08.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:25:08.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.156252 2021-05-12T12:25:08.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:25:08.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:25:08.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2021-05-12T12:25:08.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:25:08.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-fast-mark-down 0 2021-05-12T12:25:08.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T12:25:08.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:25:08.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T12:25:08.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:08.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:08.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:08.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:08.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:08.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:08.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:25:08.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:25:08.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:25:08.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:25:08.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:25:08.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:25:08.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:08.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:25:08.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:25:08.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:08.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:25:08.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:25:08.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T12:25:08.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:25:08.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:08.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156252 2021-05-12T12:25:08.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.156252 2021-05-12T12:25:08.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:25:08.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:25:08.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:25:08.920 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:25:08.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:25:08.984 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2021-05-12T12:25:08.985 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2021-05-12T12:25:09.045 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:25:09.053 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:25:09.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:25:09.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:25:09.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:25:09.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:25:09.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:25:09.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:25:09.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:25:09.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:25:09.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-force-create-pg 2021-05-12T12:25:09.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2021-05-12T12:25:09.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2021-05-12T12:25:09.056 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2021-05-12T12:25:09.056 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2021-05-12T12:25:09.056 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2021-05-12T12:25:09.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2021-05-12T12:25:09.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none ' 2021-05-12T12:25:09.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:09.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2021-05-12T12:25:09.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:25:09.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2021-05-12T12:25:09.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2021-05-12T12:25:09.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: TEST_reuse_id td/osd-force-create-pg 2021-05-12T12:25:09.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:20: TEST_reuse_id: local dir=td/osd-force-create-pg 2021-05-12T12:25:09.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: setup td/osd-force-create-pg 2021-05-12T12:25:09.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2021-05-12T12:25:09.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2021-05-12T12:25:09.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-05-12T12:25:09.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:25:09.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-05-12T12:25:09.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:09.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:09.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:09.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:09.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:09.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:09.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:25:09.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:25:09.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:25:09.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:25:09.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:25:09.067 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:25:09.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:09.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:25:09.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:25:09.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:09.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:25:09.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:25:09.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-05-12T12:25:09.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:25:09.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:09.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:09.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161013 2021-05-12T12:25:09.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:25:09.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:25:09.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2021-05-12T12:25:09.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:25:09.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:09.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:09.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.161013 2021-05-12T12:25:09.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:25:09.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:25:09.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:25:09.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:25:09.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2021-05-12T12:25:09.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:23: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-05-12T12:25:09.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2021-05-12T12:25:09.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:25:09.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:25:09.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:25:09.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2021-05-12T12:25:09.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-05-12T12:25:09.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:25:09.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:09.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:09.440 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:09.440 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:09.440 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:09.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:09.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-05-12T12:25:09.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:25:09.667 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:25:09.667 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:25:09.668 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:25:09.668 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:25:09.668 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:25:09.668 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:25:09.669 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:25:09.669 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:25:09.669 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:25:09.669 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:09.669 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:09.670 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.161013/ceph-mon.a.asok 2021-05-12T12:25:09.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:25:09.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.161013/ceph-mon.a.asok config get fsid 2021-05-12T12:25:09.761 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:25:09.761 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:25:09.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:25:09.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:25:09.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:25:09.762 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:25:09.763 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:25:09.763 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:25:09.763 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:25:09.763 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:09.764 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:09.764 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.161013/ceph-mon.a.asok 2021-05-12T12:25:09.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:25:09.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.161013/ceph-mon.a.asok config get mon_host 2021-05-12T12:25:09.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2021-05-12T12:25:09.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2021-05-12T12:25:09.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:25:09.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:25:09.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:25:09.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2021-05-12T12:25:09.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:25:10.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:25:10.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:10.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:10.719 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:10.720 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:10.720 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:10.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:10.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:25:10.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:25:10.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2021-05-12T12:25:10.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:10.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:25:10.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:25:10.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:25:10.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2021-05-12T12:25:10.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:10.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:10.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:10.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:10.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-05-12T12:25:10.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-05-12T12:25:10.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:10.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:25:10.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:10.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:25:10.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:10.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:10.748 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:10.748 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:10.748 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:10.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:10.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:10.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:10.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:25:10.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:25:10.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:10.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:10.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:10.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:10.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:10.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:25:10.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:25:10.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-force-create-pg/0 2021-05-12T12:25:10.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:25:10.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=c7922a0e-de05-433a-af0f-30249561f496 2021-05-12T12:25:10.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 c7922a0e-de05-433a-af0f-30249561f496' 2021-05-12T12:25:10.752 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 c7922a0e-de05-433a-af0f-30249561f496 2021-05-12T12:25:10.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:25:10.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAmyZtgwSvrLBAAbedBuKC4sFHZP5ezDvfH7g== 2021-05-12T12:25:10.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAmyZtgwSvrLBAAbedBuKC4sFHZP5ezDvfH7g=="}' 2021-05-12T12:25:10.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new c7922a0e-de05-433a-af0f-30249561f496 -i td/osd-force-create-pg/0/new.json 2021-05-12T12:25:11.073 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:25:11.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-force-create-pg/0/new.json 2021-05-12T12:25:11.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAmyZtgwSvrLBAAbedBuKC4sFHZP5ezDvfH7g== --osd-uuid c7922a0e-de05-433a-af0f-30249561f496 2021-05-12T12:25:11.105 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:11.102+0000 7f894acd50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:11.107 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:11.104+0000 7f894acd50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:11.108 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:11.105+0000 7f894acd50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:11.109 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:11.106+0000 7f894acd50c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-05-12T12:25:11.110 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:11.106+0000 7f894acd50c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-05-12T12:25:11.110 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:11.106+0000 7f894acd50c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-05-12T12:25:11.147 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:11.144+0000 7f894acd50c0 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2021-05-12T12:25:15.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2021-05-12T12:25:15.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:25:15.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:25:15.726 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:25:15.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:25:16.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:25:16.200 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:25:16.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:16.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:25:16.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:25:16.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:25:16.218 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:16.215+0000 7ff93b6710c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:16.220 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:16.217+0000 7ff93b6710c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:16.221 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:16.218+0000 7ff93b6710c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:16.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:25:16.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:16.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:25:16.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:16.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:16.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:16.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:16.557 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:25:16.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:16.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:25:16.737 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:16.734+0000 7ff93b6710c0 -1 Falling back to public interface 2021-05-12T12:25:16.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:17.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:17.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:17.916 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:25:17.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:17.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:17.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:25:18.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:18.965 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:18.962+0000 7ff93b6710c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:25:19.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:19.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:19.275 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:25:19.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:25:19.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:19.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:25:19.664 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/162133,v1:127.0.0.1:6803/162133] [v2:127.0.0.1:6804/162133,v1:127.0.0.1:6805/162133] exists,up c7922a0e-de05-433a-af0f-30249561f496 2021-05-12T12:25:19.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:25:19.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:25:19.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:25:19.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2021-05-12T12:25:19.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:19.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:25:19.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:25:19.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:25:19.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2021-05-12T12:25:19.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:19.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:19.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:19.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:19.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-05-12T12:25:19.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-05-12T12:25:19.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:19.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:25:19.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:19.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:25:19.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:19.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:19.682 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:19.682 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:19.683 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:19.683 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:19.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:19.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:19.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:25:19.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:25:19.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:19.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:19.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:19.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:19.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:19.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:25:19.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:25:19.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-force-create-pg/1 2021-05-12T12:25:19.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:25:19.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=6a222602-b7df-4dd4-9e67-c77bf3897acd 2021-05-12T12:25:19.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 6a222602-b7df-4dd4-9e67-c77bf3897acd' 2021-05-12T12:25:19.690 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 6a222602-b7df-4dd4-9e67-c77bf3897acd 2021-05-12T12:25:19.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:25:19.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAvyZtgaC9YKBAAp61+OShy3LK3vXxwUphVGQ== 2021-05-12T12:25:19.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAvyZtgaC9YKBAAp61+OShy3LK3vXxwUphVGQ=="}' 2021-05-12T12:25:19.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 6a222602-b7df-4dd4-9e67-c77bf3897acd -i td/osd-force-create-pg/1/new.json 2021-05-12T12:25:20.167 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:25:20.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-force-create-pg/1/new.json 2021-05-12T12:25:20.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAvyZtgaC9YKBAAp61+OShy3LK3vXxwUphVGQ== --osd-uuid 6a222602-b7df-4dd4-9e67-c77bf3897acd 2021-05-12T12:25:20.200 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:20.197+0000 7f04503aa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:20.202 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:20.199+0000 7f04503aa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:20.204 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:20.201+0000 7f04503aa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:20.204 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:20.201+0000 7f04503aa0c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-05-12T12:25:20.205 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:20.201+0000 7f04503aa0c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-05-12T12:25:20.205 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:20.201+0000 7f04503aa0c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-05-12T12:25:20.252 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:20.250+0000 7f04503aa0c0 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2021-05-12T12:25:25.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2021-05-12T12:25:25.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:25:25.085 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:25:25.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:25:25.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:25:25.573 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:25:25.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:25:25.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:25.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:25:25.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:25:25.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:25:25.593 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:25.590+0000 7fbc69f260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:25.595 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:25.592+0000 7fbc69f260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:25.597 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:25.593+0000 7fbc69f260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:25.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:25:25.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:25.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:25:25.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:25.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:25.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:25.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:25.946 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:25:25.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:25.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:25:26.112 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:26.109+0000 7fbc69f260c0 -1 Falling back to public interface 2021-05-12T12:25:26.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:27.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:27.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:27.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:27.306 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:25:27.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:27.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:25:27.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:28.306 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:28.303+0000 7fbc69f260c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:25:28.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:28.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:28.672 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:25:28.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:25:28.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:28.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:25:29.035 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/163526,v1:127.0.0.1:6811/163526] [v2:127.0.0.1:6812/163526,v1:127.0.0.1:6813/163526] exists,up 6a222602-b7df-4dd4-9e67-c77bf3897acd 2021-05-12T12:25:29.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:25:29.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:25:29.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:25:29.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2021-05-12T12:25:29.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:29.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:25:29.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:25:29.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:25:29.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2021-05-12T12:25:29.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:29.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:29.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:29.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:29.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-05-12T12:25:29.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-05-12T12:25:29.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:29.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:25:29.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:29.043 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:25:29.044 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:29.044 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:29.044 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:29.044 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:29.044 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:29.045 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:29.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:29.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:29.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:25:29.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:25:29.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:29.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:29.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:29.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:29.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:29.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:25:29.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:25:29.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-force-create-pg/2 2021-05-12T12:25:29.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:25:29.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=c395f35f-59f0-4d93-a7ca-243029c56a03 2021-05-12T12:25:29.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 c395f35f-59f0-4d93-a7ca-243029c56a03' 2021-05-12T12:25:29.048 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 c395f35f-59f0-4d93-a7ca-243029c56a03 2021-05-12T12:25:29.048 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:25:29.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA5yZtgwQD5AhAA7ppAskHMa3xOBVoiSCKAgQ== 2021-05-12T12:25:29.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA5yZtgwQD5AhAA7ppAskHMa3xOBVoiSCKAgQ=="}' 2021-05-12T12:25:29.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new c395f35f-59f0-4d93-a7ca-243029c56a03 -i td/osd-force-create-pg/2/new.json 2021-05-12T12:25:29.602 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:25:29.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-force-create-pg/2/new.json 2021-05-12T12:25:29.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA5yZtgwQD5AhAA7ppAskHMa3xOBVoiSCKAgQ== --osd-uuid c395f35f-59f0-4d93-a7ca-243029c56a03 2021-05-12T12:25:29.634 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:29.631+0000 7f060117c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:29.635 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:29.632+0000 7f060117c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:29.638 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:29.635+0000 7f060117c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:29.638 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:29.635+0000 7f060117c0c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-05-12T12:25:29.639 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:29.635+0000 7f060117c0c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-05-12T12:25:29.639 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:29.635+0000 7f060117c0c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-05-12T12:25:29.736 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:29.733+0000 7f060117c0c0 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2021-05-12T12:25:34.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2021-05-12T12:25:34.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:25:34.301 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:25:34.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:25:34.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:25:34.798 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:25:34.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:25:34.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:34.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:25:34.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:25:34.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:25:34.818 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:34.815+0000 7f1e5a9f00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:34.845 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:34.842+0000 7f1e5a9f00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:34.847 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:34.844+0000 7f1e5a9f00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:35.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:25:35.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:35.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:25:35.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:35.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:35.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:35.158 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:25:35.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:35.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:35.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:35.366 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:35.363+0000 7f1e5a9f00c0 -1 Falling back to public interface 2021-05-12T12:25:35.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:36.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:36.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:36.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:36.543 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:25:36.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:36.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:36.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:37.624 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:37.621+0000 7f1e5a9f00c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:25:37.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:37.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:37.910 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:25:37.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:25:37.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:37.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:38.280 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/164911,v1:127.0.0.1:6819/164911] [v2:127.0.0.1:6820/164911,v1:127.0.0.1:6821/164911] exists,up c395f35f-59f0-4d93-a7ca-243029c56a03 2021-05-12T12:25:38.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:25:38.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:25:38.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:25:38.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:29: TEST_reuse_id: ceph osd pool create foo 50 2021-05-12T12:25:38.949 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'foo' already exists 2021-05-12T12:25:38.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: wait_for_clean 2021-05-12T12:25:38.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:25:38.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:25:38.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:25:38.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:25:38.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:25:38.961 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:25:38.961 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:25:38.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:25:38.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:25:38.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:25:39.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:25:39.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:25:39.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:25:39.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:25:39.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:25:39.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:25:39.412 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:25:39.413 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:25:39.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:25:39.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:25:39.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:25:39.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836486 2021-05-12T12:25:39.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836486 2021-05-12T12:25:39.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836486' 2021-05-12T12:25:39.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:25:39.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:25:39.696 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705669 2021-05-12T12:25:39.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705669 2021-05-12T12:25:39.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836486 1-38654705669' 2021-05-12T12:25:39.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:25:39.698 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:25:39.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542148 2021-05-12T12:25:39.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542148 2021-05-12T12:25:39.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836486 1-38654705669 2-60129542148' 2021-05-12T12:25:39.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:25:39.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836486 2021-05-12T12:25:39.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:25:39.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:25:39.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836486 2021-05-12T12:25:39.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:25:39.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836486 2021-05-12T12:25:39.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2021-05-12T12:25:39.937 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836486 2021-05-12T12:25:39.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:25:40.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836484 -lt 21474836486 2021-05-12T12:25:40.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:25:41.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:25:41.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:25:41.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836487 -lt 21474836486 2021-05-12T12:25:41.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:25:41.684 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705669 2021-05-12T12:25:41.684 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:25:41.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:25:41.685 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705669 2021-05-12T12:25:41.686 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:25:41.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705669 2021-05-12T12:25:41.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2021-05-12T12:25:41.687 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 38654705669 2021-05-12T12:25:41.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:25:42.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705669 -lt 38654705669 2021-05-12T12:25:42.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:25:42.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542148 2021-05-12T12:25:42.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:25:42.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:25:42.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542148 2021-05-12T12:25:42.067 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:25:42.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542148 2021-05-12T12:25:42.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2021-05-12T12:25:42.068 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542148 2021-05-12T12:25:42.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:25:42.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542148 -lt 60129542148 2021-05-12T12:25:42.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:25:42.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:25:42.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:25:42.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 50 == 0 2021-05-12T12:25:42.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:25:42.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:25:42.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:25:42.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:25:42.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:25:42.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:25:42.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:25:43.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=27 2021-05-12T12:25:43.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:25:43.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:25:43.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:25:43.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 27 = 50 2021-05-12T12:25:43.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 27 '!=' -1 2021-05-12T12:25:43.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:25:43.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=27 2021-05-12T12:25:43.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:25:43.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:25:43.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:25:43.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:25:43.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:25:43.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:25:43.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:25:43.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:25:43.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:25:43.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:25:44.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=27 2021-05-12T12:25:44.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:25:44.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:25:44.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:25:44.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 27 = 50 2021-05-12T12:25:44.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 27 '!=' 27 2021-05-12T12:25:44.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:25:44.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:25:44.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:25:44.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:25:44.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:25:44.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:25:44.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:25:45.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:25:45.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:25:45.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 13 )) 2021-05-12T12:25:45.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:25:45.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:25:45.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:25:45.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:25:45.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:25:45.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:25:45.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:25:45.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:25:45.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:25:45.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:25:45.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=50 2021-05-12T12:25:45.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:25:45.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:25:45.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:25:46.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 50 = 50 2021-05-12T12:25:46.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:25:46.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:25:46.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:32: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2021-05-12T12:25:46.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:46.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:46.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:46.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:46.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:46.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:46.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2021-05-12T12:25:46.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:46.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:46.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:46.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:46.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:46.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:46.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2021-05-12T12:25:46.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:46.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:46.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:46.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:46.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:46.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:46.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2021-05-12T12:25:47.554 INFO:tasks.workunit.client.0.smithi191.stderr:PG '1.0' not found 2021-05-12T12:25:47.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2021-05-12T12:25:48.825 INFO:tasks.workunit.client.0.smithi191.stdout: marking collection for removal 2021-05-12T12:25:48.825 INFO:tasks.workunit.client.0.smithi191.stdout:setting '_remove' omap key 2021-05-12T12:25:48.826 INFO:tasks.workunit.client.0.smithi191.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-05-12T12:25:48.899 INFO:tasks.workunit.client.0.smithi191.stdout:Remove successful 2021-05-12T12:25:49.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2021-05-12T12:25:50.528 INFO:tasks.workunit.client.0.smithi191.stderr:PG '1.0' not found 2021-05-12T12:25:50.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2021-05-12T12:25:50.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:50.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:25:50.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:25:50.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:25:50.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-force-create-pg/0 2021-05-12T12:25:50.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:50.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:50.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:50.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:50.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-05-12T12:25:50.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-05-12T12:25:50.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:50.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:25:50.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:50.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:25:50.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:50.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:50.807 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:50.808 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:50.808 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:50.808 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:50.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:50.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:50.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:50.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:50.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:50.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:50.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:50.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:25:50.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:25:50.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-force-create-pg/0 2021-05-12T12:25:50.811 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:25:50.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:25:50.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:50.812 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-force-create-pg/0/whoami 2021-05-12T12:25:50.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:25:50.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:25:50.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:25:50.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:25:50.824 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:50.821+0000 7f43f3ead0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:50.826 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:50.823+0000 7f43f3ead0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:50.828 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:50.825+0000 7f43f3ead0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:51.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:25:51.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:51.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:25:51.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:51.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:51.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:51.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:51.171 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:25:51.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:51.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:25:51.348 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:51.345+0000 7f43f3ead0c0 -1 Falling back to public interface 2021-05-12T12:25:51.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:52.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:52.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:52.551 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:25:52.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:52.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:52.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:25:52.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:53.690 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:53.686+0000 7f43f3ead0c0 -1 osd.0 21 log_to_monitors {default=true} 2021-05-12T12:25:53.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:53.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:53.916 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:25:53.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:25:53.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:53.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:25:54.304 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 25 up_thru 16 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/167198,v1:127.0.0.1:6803/167198] [v2:127.0.0.1:6804/167198,v1:127.0.0.1:6805/167198] exists,up c7922a0e-de05-433a-af0f-30249561f496 2021-05-12T12:25:54.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:25:54.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:25:54.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:25:54.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2021-05-12T12:25:54.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:54.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:25:54.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:25:54.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:25:54.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-force-create-pg/1 2021-05-12T12:25:54.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:54.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:54.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:54.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:54.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-05-12T12:25:54.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-05-12T12:25:54.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:54.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:25:54.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:54.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:25:54.317 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:54.318 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:54.319 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:54.319 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:54.319 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:54.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:54.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:54.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:54.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:54.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:54.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:54.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:54.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:54.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:25:54.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:25:54.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-force-create-pg/1 2021-05-12T12:25:54.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:25:54.327 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:25:54.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:54.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-force-create-pg/1/whoami 2021-05-12T12:25:54.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:25:54.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:25:54.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:25:54.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:25:54.334 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:54.322+0000 7fe499aaa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:54.399 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:54.395+0000 7fe499aaa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:54.400 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:54.397+0000 7fe499aaa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:54.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:25:54.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:54.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:25:54.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:54.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:54.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:54.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:54.720 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:25:54.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:54.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:25:54.917 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:54.914+0000 7fe499aaa0c0 -1 Falling back to public interface 2021-05-12T12:25:55.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:56.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:56.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:56.075 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:25:56.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:56.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:56.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:25:56.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:57.272 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:57.269+0000 7fe499aaa0c0 -1 osd.1 21 log_to_monitors {default=true} 2021-05-12T12:25:57.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:57.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:57.441 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:25:57.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:25:57.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:57.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:25:57.854 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 28 up_thru 16 down_at 23 last_clean_interval [9,21) [v2:127.0.0.1:6810/167773,v1:127.0.0.1:6811/167773] [v2:127.0.0.1:6812/167773,v1:127.0.0.1:6813/167773] exists,up 6a222602-b7df-4dd4-9e67-c77bf3897acd 2021-05-12T12:25:57.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:25:57.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:25:57.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:25:57.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2021-05-12T12:25:57.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:57.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:25:57.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:25:57.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:25:57.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-force-create-pg/2 2021-05-12T12:25:57.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:57.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:57.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:57.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:57.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-05-12T12:25:57.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-05-12T12:25:57.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:57.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:25:57.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:57.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:25:57.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:57.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:57.868 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:57.868 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:57.868 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:25:57.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:57.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' 2021-05-12T12:25:57.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:57.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:57.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:57.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:57.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:57.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:57.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:25:57.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:25:57.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-force-create-pg/2 2021-05-12T12:25:57.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:25:57.872 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:25:57.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=c42780d0-c28a-4d3f-a3e6-7e2070b07e89 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.161013/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:57.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-force-create-pg/2/whoami 2021-05-12T12:25:57.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:25:57.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:25:57.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:25:57.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:25:57.876 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:57.872+0000 7f0874da50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:57.877 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:57.874+0000 7f0874da50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:57.878 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:57.875+0000 7f0874da50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:58.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:25:58.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:58.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:25:58.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:58.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:58.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:58.235 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:25:58.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:58.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:58.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:58.393 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:25:58.390+0000 7f0874da50c0 -1 Falling back to public interface 2021-05-12T12:25:58.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:59.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:59.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:59.600 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:25:59.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:59.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:59.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:59.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:00.656 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:00.653+0000 7f0874da50c0 -1 osd.2 21 log_to_monitors {default=true} 2021-05-12T12:26:00.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:00.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:00.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:00.963 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:26:00.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:00.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:01.326 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 31 up_thru 15 down_at 26 last_clean_interval [14,21) [v2:127.0.0.1:6818/168312,v1:127.0.0.1:6819/168312] [v2:127.0.0.1:6820/168312,v1:127.0.0.1:6821/168312] exists,up c395f35f-59f0-4d93-a7ca-243029c56a03 2021-05-12T12:26:01.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:01.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:01.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:01.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: sleep 10 2021-05-12T12:26:11.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: ceph pg ls 2021-05-12T12:26:11.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep 1.0 2021-05-12T12:26:11.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep stale 2021-05-12T12:26:11.686 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 0 0 0 stale+active+clean 31s 0'0 20:10 [1]p1 [1]p1 2021-05-12T12:25:38.582851+0000 2021-05-12T12:25:38.582851+0000 2021-05-12T12:26:11.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:44: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2021-05-12T12:26:12.319 INFO:tasks.workunit.client.0.smithi191.stderr:pg 1.0 already creating 2021-05-12T12:26:12.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: wait_for_clean 2021-05-12T12:26:12.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:26:12.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:26:12.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:26:12.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:26:12.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:26:12.330 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:26:12.330 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:26:12.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:26:12.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:26:12.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:26:12.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:26:12.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:26:12.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:26:12.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:26:12.417 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:26:12.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:26:12.772 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:26:12.772 INFO:tasks.workunit.client.0.smithi191.stderr:2' 2021-05-12T12:26:12.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:26:12.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:26:12.773 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:26:12.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=107374182406 2021-05-12T12:26:12.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 107374182406 2021-05-12T12:26:12.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-107374182406' 2021-05-12T12:26:12.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:26:12.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:26:13.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=120259084294 2021-05-12T12:26:13.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 120259084294 2021-05-12T12:26:13.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-107374182406 1-120259084294' 2021-05-12T12:26:13.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:26:13.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:26:13.142 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=133143986181 2021-05-12T12:26:13.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 133143986181 2021-05-12T12:26:13.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-107374182406 1-120259084294 2-133143986181' 2021-05-12T12:26:13.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:26:13.144 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-107374182406 2021-05-12T12:26:13.144 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:26:13.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:26:13.145 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-107374182406 2021-05-12T12:26:13.145 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:26:13.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=107374182406 2021-05-12T12:26:13.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2021-05-12T12:26:13.146 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 107374182406 2021-05-12T12:26:13.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:26:13.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 107374182406 -lt 107374182406 2021-05-12T12:26:13.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:26:13.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-120259084294 2021-05-12T12:26:13.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:26:13.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:26:13.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-120259084294 2021-05-12T12:26:13.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:26:13.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=120259084294 2021-05-12T12:26:13.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 120259084294' 2021-05-12T12:26:13.505 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 120259084294 2021-05-12T12:26:13.505 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:26:13.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 120259084294 -lt 120259084294 2021-05-12T12:26:13.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:26:13.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-133143986181 2021-05-12T12:26:13.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:26:13.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:26:13.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-133143986181 2021-05-12T12:26:13.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:26:13.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=133143986181 2021-05-12T12:26:13.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 133143986181' 2021-05-12T12:26:13.882 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 133143986181 2021-05-12T12:26:13.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:26:14.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 133143986181 -lt 133143986181 2021-05-12T12:26:14.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:26:14.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:26:14.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:26:14.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 50 == 0 2021-05-12T12:26:14.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:26:14.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:26:14.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:26:14.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:26:14.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:26:14.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:26:14.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:26:15.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=50 2021-05-12T12:26:15.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:26:15.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:26:15.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:26:15.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 50 = 50 2021-05-12T12:26:15.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:26:15.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:26:15.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:26:15.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-force-create-pg 0 2021-05-12T12:26:15.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-05-12T12:26:15.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:26:15.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-05-12T12:26:15.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:26:15.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:26:15.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:26:15.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:26:15.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:26:15.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:26:15.675 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:26:15.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:26:15.676 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:26:15.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:26:15.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:26:15.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:26:15.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:26:15.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:26:15.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:26:15.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:26:15.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:26:15.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:26:15.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-05-12T12:26:15.687 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:26:15.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:15.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161013 2021-05-12T12:26:15.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161013 2021-05-12T12:26:15.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:26:15.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:26:15.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:26:15.690 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:26:15.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:26:15.754 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2021-05-12T12:26:15.755 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2021-05-12T12:26:15.814 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:26:15.821 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:26:15.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:26:15.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:26:15.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:26:15.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:26:15.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:26:15.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:26:15.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:26:15.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:26:15.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-markdown 2021-05-12T12:26:15.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2021-05-12T12:26:15.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2021-05-12T12:26:15.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2021-05-12T12:26:15.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2021-05-12T12:26:15.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2021-05-12T12:26:15.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2021-05-12T12:26:15.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none ' 2021-05-12T12:26:15.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2021-05-12T12:26:15.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2021-05-12T12:26:15.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:26:15.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2021-05-12T12:26:15.828 INFO:tasks.workunit.client.0.smithi191.stderr:TEST_markdown_boot_exceed_time 2021-05-12T12:26:15.829 INFO:tasks.workunit.client.0.smithi191.stderr:TEST_markdown_exceed_maxdown_count 2021-05-12T12:26:15.829 INFO:tasks.workunit.client.0.smithi191.stderr:TEST_osd_stop' 2021-05-12T12:26:15.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T12:26:15.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T12:26:15.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T12:26:15.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T12:26:15.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:26:15.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:26:15.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:26:15.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:26:15.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:26:15.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:26:15.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:26:15.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:26:15.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:26:15.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:26:15.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:26:15.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:26:15.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:26:15.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:26:15.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:26:15.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:26:15.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:26:15.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:26:15.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:26:15.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:26:15.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:26:15.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:26:15.837 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:26:15.837 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:15.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:26:15.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169344 2021-05-12T12:26:15.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:26:15.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:26:15.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T12:26:15.839 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:26:15.839 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:15.839 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:26:15.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.169344 2021-05-12T12:26:15.840 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:26:15.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:26:15.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:26:15.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:26:15.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T12:26:15.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2021-05-12T12:26:15.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2021-05-12T12:26:15.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2021-05-12T12:26:15.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T12:26:15.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:26:15.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:26:15.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:26:15.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T12:26:15.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T12:26:16.178 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:26:16.178 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:16.179 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:16.179 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:16.179 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:16.180 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:26:16.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:26:16.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:26:16.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:26:16.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:26:16.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:26:16.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:26:16.427 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:26:16.427 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:26:16.427 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:26:16.427 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:26:16.427 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:26:16.428 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:26:16.428 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:16.428 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:26:16.428 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169344/ceph-mon.a.asok 2021-05-12T12:26:16.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:26:16.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.169344/ceph-mon.a.asok config get fsid 2021-05-12T12:26:16.517 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:26:16.517 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:26:16.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:26:16.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:26:16.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:26:16.519 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:26:16.519 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:26:16.519 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:26:16.519 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:26:16.520 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:16.520 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:26:16.520 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169344/ceph-mon.a.asok 2021-05-12T12:26:16.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:26:16.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.169344/ceph-mon.a.asok config get mon_host 2021-05-12T12:26:16.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2021-05-12T12:26:16.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T12:26:16.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:26:16.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:26:16.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:26:16.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T12:26:16.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:26:17.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:26:17.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:17.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:17.483 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:17.483 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:17.483 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:26:17.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:26:17.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:26:17.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:26:17.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2021-05-12T12:26:17.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:26:17.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:26:17.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:26:17.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:26:17.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T12:26:17.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:26:17.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:26:17.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:26:17.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:26:17.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T12:26:17.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T12:26:17.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:26:17.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:26:17.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:26:17.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:26:17.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:17.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:17.511 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:17.512 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:17.512 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:26:17.512 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:26:17.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:26:17.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:26:17.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:26:17.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:26:17.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:26:17.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:26:17.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:26:17.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:26:17.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:17.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:26:17.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:26:17.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T12:26:17.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:26:17.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=dc14c13b-02dd-4d89-bebc-380c08cf0f5e 2021-05-12T12:26:17.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 dc14c13b-02dd-4d89-bebc-380c08cf0f5e' 2021-05-12T12:26:17.516 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 dc14c13b-02dd-4d89-bebc-380c08cf0f5e 2021-05-12T12:26:17.516 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:26:17.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBpyZtgfFXVHhAApaRBJlhaDx6+1flzP/t4Jg== 2021-05-12T12:26:17.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBpyZtgfFXVHhAApaRBJlhaDx6+1flzP/t4Jg=="}' 2021-05-12T12:26:17.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new dc14c13b-02dd-4d89-bebc-380c08cf0f5e -i td/osd-markdown/0/new.json 2021-05-12T12:26:17.837 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:26:17.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T12:26:17.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBpyZtgfFXVHhAApaRBJlhaDx6+1flzP/t4Jg== --osd-uuid dc14c13b-02dd-4d89-bebc-380c08cf0f5e 2021-05-12T12:26:17.870 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:17.867+0000 7fc45ae2d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:17.872 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:17.869+0000 7fc45ae2d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:17.873 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:17.870+0000 7fc45ae2d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:17.874 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:17.871+0000 7fc45ae2d0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:26:17.875 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:17.871+0000 7fc45ae2d0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:26:17.875 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:17.871+0000 7fc45ae2d0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:26:17.904 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:17.900+0000 7fc45ae2d0c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T12:26:22.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T12:26:22.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:26:22.308 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:26:22.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:26:22.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:26:22.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:26:22.785 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:26:22.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:26:22.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:22.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:26:22.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:26:22.804 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:22.800+0000 7fb9f54670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:22.805 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:22.802+0000 7fb9f54670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:22.807 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:22.804+0000 7fb9f54670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:23.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:26:23.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:26:23.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:26:23.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:26:23.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:26:23.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:23.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:26:23.165 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:26:23.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:23.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:26:23.324 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:23.321+0000 7fb9f54670c0 -1 Falling back to public interface 2021-05-12T12:26:23.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:24.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:24.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:24.521 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:26:24.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:26:24.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:24.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:26:24.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:25.531 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:25.526+0000 7fb9f54670c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:26:25.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:25.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:25.881 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:26:25.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:25.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:25.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:26:26.263 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/170464,v1:127.0.0.1:6803/170464] [v2:127.0.0.1:6804/170464,v1:127.0.0.1:6805/170464] exists,up dc14c13b-02dd-4d89-bebc-380c08cf0f5e 2021-05-12T12:26:26.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:26.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:26.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:26.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2021-05-12T12:26:26.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:26:26.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:26:26.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:26:26.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:26:26.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T12:26:26.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:26:26.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:26:26.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:26:26.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:26:26.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T12:26:26.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T12:26:26.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:26:26.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:26:26.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:26:26.272 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:26:26.272 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:26.272 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:26.272 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:26.272 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:26.273 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:26:26.273 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:26:26.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:26:26.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:26:26.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:26:26.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:26:26.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:26:26.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:26:26.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:26:26.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:26:26.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:26.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:26:26.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:26:26.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T12:26:26.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:26:26.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=73c1d633-85f8-4178-aab3-98a3665ca180 2021-05-12T12:26:26.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 73c1d633-85f8-4178-aab3-98a3665ca180' 2021-05-12T12:26:26.277 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 73c1d633-85f8-4178-aab3-98a3665ca180 2021-05-12T12:26:26.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:26:26.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQByyZtggahtEBAA4rR+WSQ1oyUSAJRK18EweQ== 2021-05-12T12:26:26.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQByyZtggahtEBAA4rR+WSQ1oyUSAJRK18EweQ=="}' 2021-05-12T12:26:26.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 73c1d633-85f8-4178-aab3-98a3665ca180 -i td/osd-markdown/1/new.json 2021-05-12T12:26:26.784 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:26:26.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T12:26:26.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQByyZtggahtEBAA4rR+WSQ1oyUSAJRK18EweQ== --osd-uuid 73c1d633-85f8-4178-aab3-98a3665ca180 2021-05-12T12:26:26.817 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:26.814+0000 7f227bca70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:26.835 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:26.831+0000 7f227bca70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:26.836 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:26.833+0000 7f227bca70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:26.838 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:26.834+0000 7f227bca70c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:26:26.838 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:26.834+0000 7f227bca70c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:26:26.838 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:26.834+0000 7f227bca70c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:26:26.926 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:26.922+0000 7f227bca70c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T12:26:31.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T12:26:31.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:26:31.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:26:31.484 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:26:31.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:26:31.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:26:31.969 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:26:31.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:31.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:26:31.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:26:31.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:26:31.988 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:31.984+0000 7f4d963fa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:31.989 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:31.986+0000 7f4d963fa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:31.991 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:31.988+0000 7f4d963fa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:32.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:26:32.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:26:32.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:26:32.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:26:32.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:26:32.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:32.331 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:26:32.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:26:32.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:32.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:26:32.507 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:32.504+0000 7f4d963fa0c0 -1 Falling back to public interface 2021-05-12T12:26:32.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:33.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:33.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:33.686 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:26:33.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:26:33.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:33.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:26:34.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:34.714 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:34.711+0000 7f4d963fa0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:26:35.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:35.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:35.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:35.053 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:26:35.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:35.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:26:35.422 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/171857,v1:127.0.0.1:6811/171857] [v2:127.0.0.1:6812/171857,v1:127.0.0.1:6813/171857] exists,up 73c1d633-85f8-4178-aab3-98a3665ca180 2021-05-12T12:26:35.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:35.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:35.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:35.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2021-05-12T12:26:35.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:26:35.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:26:35.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:26:35.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:26:35.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T12:26:35.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:26:35.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:26:35.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:26:35.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:26:35.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T12:26:35.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T12:26:35.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:26:35.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:26:35.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:26:35.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:26:35.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:35.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:35.432 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:35.433 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:35.434 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:26:35.434 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:26:35.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:26:35.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:26:35.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:26:35.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:26:35.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:26:35.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:26:35.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:26:35.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:26:35.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:35.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:26:35.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:26:35.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T12:26:35.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:26:35.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=285b2c14-14ca-4e01-a066-84e8b3306cf5 2021-05-12T12:26:35.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 285b2c14-14ca-4e01-a066-84e8b3306cf5' 2021-05-12T12:26:35.441 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 285b2c14-14ca-4e01-a066-84e8b3306cf5 2021-05-12T12:26:35.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:26:35.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB7yZtggn4FGhAANReE0bhjS/L7zjHXEeNHGA== 2021-05-12T12:26:35.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB7yZtggn4FGhAANReE0bhjS/L7zjHXEeNHGA=="}' 2021-05-12T12:26:35.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 285b2c14-14ca-4e01-a066-84e8b3306cf5 -i td/osd-markdown/2/new.json 2021-05-12T12:26:35.971 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:26:35.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T12:26:35.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB7yZtggn4FGhAANReE0bhjS/L7zjHXEeNHGA== --osd-uuid 285b2c14-14ca-4e01-a066-84e8b3306cf5 2021-05-12T12:26:36.003 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:36.000+0000 7f643f7fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:36.021 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:36.017+0000 7f643f7fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:36.022 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:36.019+0000 7f643f7fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:36.023 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:36.020+0000 7f643f7fc0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:26:36.023 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:36.020+0000 7f643f7fc0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:26:36.024 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:36.020+0000 7f643f7fc0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:26:36.097 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:36.094+0000 7f643f7fc0c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T12:26:40.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T12:26:40.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:26:40.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:26:40.899 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:26:40.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:26:41.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:26:41.380 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:26:41.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:41.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:26:41.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:26:41.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:26:41.402 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:41.395+0000 7f663f1680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:41.402 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:41.397+0000 7f663f1680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:41.402 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:41.398+0000 7f663f1680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:41.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:26:41.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:26:41.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:26:41.739 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:26:41.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:26:41.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:41.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:26:41.741 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:26:41.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:41.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:41.919 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:41.916+0000 7f663f1680c0 -1 Falling back to public interface 2021-05-12T12:26:42.141 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:43.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:43.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:43.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:26:43.144 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:26:43.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:43.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:43.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:44.124 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:26:44.121+0000 7f663f1680c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:26:44.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:44.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:44.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:44.506 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:26:44.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:44.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:44.905 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/173242,v1:127.0.0.1:6819/173242] [v2:127.0.0.1:6820/173242,v1:127.0.0.1:6821/173242] exists,up 285b2c14-14ca-4e01-a066-84e8b3306cf5 2021-05-12T12:26:44.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:44.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:44.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:44.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2021-05-12T12:26:44.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:26:45.330 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' does not exist 2021-05-12T12:26:45.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:26:45.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:26:46.017 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' already exists 2021-05-12T12:26:46.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:26:47.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:26:48.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2021-05-12T12:26:48.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2021-05-12T12:26:48.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2021-05-12T12:26:48.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-05-12T12:26:48.464 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:26:48.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2021-05-12T12:26:48.617 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:26:48.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2021-05-12T12:26:48.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2021-05-12T12:26:48.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2021-05-12T12:26:48.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-05-12T12:26:48.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2021-05-12T12:26:48.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:26:48.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:26:48.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:26:49.090 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:26:49.091 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:26:49.091 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:26:49.091 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:26:49.092 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:26:49.092 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:26:49.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:26:49.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:26:49.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:26:49.456 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:26:49.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:26:49.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:26:49.873 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:26:49.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:26:59.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:26:59.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:27:00.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:27:00.380 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:27:00.380 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:27:00.380 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:27:00.380 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:00.381 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:27:00.381 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:27:00.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:27:00.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:27:00.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:27:00.745 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:00.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:27:00.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:27:01.127 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:27:01.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:27:11.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:27:11.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:27:11.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:27:11.614 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:27:11.614 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:27:11.614 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:27:11.615 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:11.615 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:27:11.615 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:27:11.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:27:11.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:27:11.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:27:11.995 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:11.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:27:11.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:27:12.438 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:27:12.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:27:22.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2021-05-12T12:27:37.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:27:37.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2021-05-12T12:27:37.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2021-05-12T12:27:37.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2021-05-12T12:27:37.941 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:37.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T12:27:37.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:27:37.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:27:37.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:27:37.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:27:37.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:27:37.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:27:37.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:27:37.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:27:38.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:27:38.050 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:27:38.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:27:38.051 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:27:38.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:27:38.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:27:38.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:27:38.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:27:38.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:27:38.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:27:38.054 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:27:38.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:27:38.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:27:38.056 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:27:38.061 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:27:38.062 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:38.062 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:38.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169344 2021-05-12T12:27:38.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:27:38.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:27:38.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T12:27:38.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T12:27:38.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T12:27:38.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T12:27:38.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:27:38.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:27:38.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:27:38.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:27:38.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:27:38.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:27:38.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:27:38.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:27:38.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:27:38.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:27:38.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:27:38.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:27:38.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:27:38.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:27:38.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:27:38.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:27:38.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:27:38.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:27:38.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:27:38.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:27:38.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:27:38.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:27:38.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:27:38.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:38.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:38.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169344 2021-05-12T12:27:38.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:27:38.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:27:38.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T12:27:38.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:27:38.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:38.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:38.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.169344 2021-05-12T12:27:38.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:27:38.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:27:38.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:27:38.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T12:27:38.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2021-05-12T12:27:38.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2021-05-12T12:27:38.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2021-05-12T12:27:38.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T12:27:38.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:27:38.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:27:38.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:27:38.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T12:27:38.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T12:27:38.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:27:38.436 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:27:38.436 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:27:38.436 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:27:38.436 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:38.437 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:38.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:27:38.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:27:38.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:27:38.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:27:38.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:27:38.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:27:38.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:27:38.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:27:38.682 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:27:38.682 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:27:38.682 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:27:38.682 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:27:38.683 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:38.683 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:38.683 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169344/ceph-mon.a.asok 2021-05-12T12:27:38.683 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:27:38.683 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.169344/ceph-mon.a.asok config get fsid 2021-05-12T12:27:38.778 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:27:38.778 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:27:38.779 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:27:38.779 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:27:38.779 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:27:38.779 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:27:38.780 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:27:38.780 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:27:38.780 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:27:38.780 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:38.781 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:38.781 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169344/ceph-mon.a.asok 2021-05-12T12:27:38.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:27:38.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.169344/ceph-mon.a.asok config get mon_host 2021-05-12T12:27:38.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2021-05-12T12:27:38.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T12:27:38.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:27:38.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:27:38.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:27:38.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T12:27:38.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:27:39.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:27:39.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:27:39.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:27:39.366 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:27:39.366 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:39.366 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:39.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:27:39.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:27:39.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:27:39.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2021-05-12T12:27:39.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:27:39.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:27:39.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:27:39.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:27:39.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T12:27:39.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:27:39.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:27:39.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:27:39.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:27:39.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T12:27:39.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T12:27:39.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:27:39.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:27:39.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:27:39.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:27:39.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:27:39.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:27:39.395 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:27:39.395 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:39.395 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:39.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:27:39.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:27:39.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:27:39.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:27:39.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:27:39.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:27:39.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:27:39.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:27:39.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:27:39.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:27:39.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:27:39.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:27:39.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T12:27:39.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:27:39.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=fe79ab49-2d2d-4d90-8553-851f40ed0daa 2021-05-12T12:27:39.399 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 fe79ab49-2d2d-4d90-8553-851f40ed0daa 2021-05-12T12:27:39.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 fe79ab49-2d2d-4d90-8553-851f40ed0daa' 2021-05-12T12:27:39.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:27:39.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC7yZtgTvneFxAA7z0+xdctHJYdYHk4SB0NgA== 2021-05-12T12:27:39.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC7yZtgTvneFxAA7z0+xdctHJYdYHk4SB0NgA=="}' 2021-05-12T12:27:39.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new fe79ab49-2d2d-4d90-8553-851f40ed0daa -i td/osd-markdown/0/new.json 2021-05-12T12:27:39.711 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:27:39.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T12:27:39.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC7yZtgTvneFxAA7z0+xdctHJYdYHk4SB0NgA== --osd-uuid fe79ab49-2d2d-4d90-8553-851f40ed0daa 2021-05-12T12:27:39.744 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:39.740+0000 7f746537f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:39.745 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:39.742+0000 7f746537f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:39.747 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:39.744+0000 7f746537f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:39.748 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:39.745+0000 7f746537f0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:27:39.748 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:39.745+0000 7f746537f0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:27:39.749 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:39.745+0000 7f746537f0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:27:39.804 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:39.799+0000 7f746537f0c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T12:27:44.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T12:27:44.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:27:44.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:27:44.345 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:27:44.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:27:44.849 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:27:44.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:27:44.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:27:44.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:27:44.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:27:44.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:27:44.869 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:44.865+0000 7f46d574e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:44.870 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:44.867+0000 7f46d574e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:44.871 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:44.868+0000 7f46d574e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:45.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:27:45.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:27:45.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:27:45.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:27:45.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:27:45.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:27:45.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:27:45.205 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:27:45.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:27:45.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:27:45.395 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:45.392+0000 7f46d574e0c0 -1 Falling back to public interface 2021-05-12T12:27:45.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:27:46.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:27:46.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:27:46.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:27:46.581 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:27:46.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:27:46.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:27:46.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:27:47.596 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:47.592+0000 7f46d574e0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:27:47.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:27:47.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:27:47.946 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:27:47.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:27:47.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:27:47.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:27:48.318 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/175402,v1:127.0.0.1:6803/175402] [v2:127.0.0.1:6804/175402,v1:127.0.0.1:6805/175402] exists,up fe79ab49-2d2d-4d90-8553-851f40ed0daa 2021-05-12T12:27:48.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:27:48.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:27:48.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:27:48.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2021-05-12T12:27:48.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:27:48.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:27:48.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:27:48.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:27:48.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T12:27:48.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:27:48.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:27:48.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:27:48.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:27:48.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T12:27:48.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T12:27:48.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:27:48.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:27:48.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:27:48.327 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:27:48.328 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:27:48.328 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:27:48.328 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:27:48.328 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:48.328 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:48.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:27:48.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:27:48.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:27:48.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:27:48.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:27:48.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:27:48.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:27:48.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:27:48.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:27:48.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:27:48.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:27:48.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:27:48.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T12:27:48.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:27:48.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=a0d01c8b-60c4-4d5d-97e9-3caef6ad595f 2021-05-12T12:27:48.332 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 a0d01c8b-60c4-4d5d-97e9-3caef6ad595f 2021-05-12T12:27:48.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 a0d01c8b-60c4-4d5d-97e9-3caef6ad595f' 2021-05-12T12:27:48.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:27:48.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDEyZtgii3JExAAbZle4yTlgk/tJK862mft0Q== 2021-05-12T12:27:48.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDEyZtgii3JExAAbZle4yTlgk/tJK862mft0Q=="}' 2021-05-12T12:27:48.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new a0d01c8b-60c4-4d5d-97e9-3caef6ad595f -i td/osd-markdown/1/new.json 2021-05-12T12:27:48.906 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:27:48.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T12:27:48.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDEyZtgii3JExAAbZle4yTlgk/tJK862mft0Q== --osd-uuid a0d01c8b-60c4-4d5d-97e9-3caef6ad595f 2021-05-12T12:27:48.939 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:48.936+0000 7fcbdc3430c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:48.942 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:48.938+0000 7fcbdc3430c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:48.943 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:48.940+0000 7fcbdc3430c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:48.944 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:48.941+0000 7fcbdc3430c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:27:48.944 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:48.941+0000 7fcbdc3430c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:27:48.944 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:48.941+0000 7fcbdc3430c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:27:49.042 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:49.038+0000 7fcbdc3430c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T12:27:53.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T12:27:53.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:27:53.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:27:53.719 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:27:53.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:27:54.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:27:54.204 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:27:54.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:27:54.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:27:54.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:27:54.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:27:54.222 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:54.219+0000 7f9c3271b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:54.224 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:54.221+0000 7f9c3271b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:54.226 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:54.223+0000 7f9c3271b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:54.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:27:54.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:27:54.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:27:54.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:27:54.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:27:54.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:27:54.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:27:54.581 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:27:54.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:27:54.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:27:54.742 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:54.739+0000 7f9c3271b0c0 -1 Falling back to public interface 2021-05-12T12:27:54.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:27:55.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:27:55.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:27:55.941 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:27:55.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:27:55.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:27:55.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:27:56.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:27:56.937 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:56.934+0000 7f9c3271b0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:27:57.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:27:57.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:27:57.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:27:57.310 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:27:57.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:27:57.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:27:57.680 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/176787,v1:127.0.0.1:6811/176787] [v2:127.0.0.1:6812/176787,v1:127.0.0.1:6813/176787] exists,up a0d01c8b-60c4-4d5d-97e9-3caef6ad595f 2021-05-12T12:27:57.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:27:57.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:27:57.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:27:57.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2021-05-12T12:27:57.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:27:57.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:27:57.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:27:57.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:27:57.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T12:27:57.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:27:57.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:27:57.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:27:57.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:27:57.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T12:27:57.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T12:27:57.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:27:57.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:27:57.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:27:57.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:27:57.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:27:57.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:27:57.690 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:27:57.690 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:57.690 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:27:57.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:27:57.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:27:57.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:27:57.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:27:57.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:27:57.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:27:57.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:27:57.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:27:57.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:27:57.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:27:57.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:27:57.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:27:57.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T12:27:57.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:27:57.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7d94f4a9-efab-44c3-a8b4-e31acf277db0 2021-05-12T12:27:57.695 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 7d94f4a9-efab-44c3-a8b4-e31acf277db0 2021-05-12T12:27:57.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 7d94f4a9-efab-44c3-a8b4-e31acf277db0' 2021-05-12T12:27:57.696 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:27:57.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDNyZtgxkhaKRAA3W5tuzpisW1LpDtT/+Ya+w== 2021-05-12T12:27:57.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDNyZtgxkhaKRAA3W5tuzpisW1LpDtT/+Ya+w=="}' 2021-05-12T12:27:57.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7d94f4a9-efab-44c3-a8b4-e31acf277db0 -i td/osd-markdown/2/new.json 2021-05-12T12:27:58.157 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:27:58.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T12:27:58.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDNyZtgxkhaKRAA3W5tuzpisW1LpDtT/+Ya+w== --osd-uuid 7d94f4a9-efab-44c3-a8b4-e31acf277db0 2021-05-12T12:27:58.190 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:58.186+0000 7f32037ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:58.191 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:58.188+0000 7f32037ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:58.193 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:58.190+0000 7f32037ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:58.194 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:58.190+0000 7f32037ce0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:27:58.194 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:58.190+0000 7f32037ce0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:27:58.194 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:58.190+0000 7f32037ce0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:27:58.233 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:27:58.230+0000 7f32037ce0c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T12:28:02.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T12:28:02.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:28:02.878 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:28:02.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:28:02.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:28:03.361 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:28:03.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:28:03.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:28:03.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:28:03.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:28:03.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:28:03.380 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:28:03.376+0000 7f14f01e90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:03.381 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:28:03.378+0000 7f14f01e90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:03.383 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:28:03.380+0000 7f14f01e90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:03.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:28:03.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:28:03.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:28:03.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:28:03.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:28:03.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:28:03.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:28:03.724 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:28:03.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:28:03.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:28:03.909 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:28:03.906+0000 7f14f01e90c0 -1 Falling back to public interface 2021-05-12T12:28:04.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:28:05.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:28:05.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:28:05.091 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:28:05.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:28:05.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:28:05.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:28:05.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:28:06.162 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:28:06.159+0000 7f14f01e90c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:28:06.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:28:06.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:28:06.455 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:28:06.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:28:06.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:28:06.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:28:06.837 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/178172,v1:127.0.0.1:6819/178172] [v2:127.0.0.1:6820/178172,v1:127.0.0.1:6821/178172] exists,up 7d94f4a9-efab-44c3-a8b4-e31acf277db0 2021-05-12T12:28:06.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:28:06.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:28:06.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:28:06.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2021-05-12T12:28:06.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:28:07.189 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' does not exist 2021-05-12T12:28:07.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:28:07.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:28:07.737 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' already exists 2021-05-12T12:28:07.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:28:08.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:28:10.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2021-05-12T12:28:10.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2021-05-12T12:28:10.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2021-05-12T12:28:10.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-05-12T12:28:10.219 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:28:10.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2021-05-12T12:28:10.347 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:28:10.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2021-05-12T12:28:10.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-05-12T12:28:10.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2021-05-12T12:28:10.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-05-12T12:28:10.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-05-12T12:28:10.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:28:10.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:28:10.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:28:10.854 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:28:10.854 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:28:10.854 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:28:10.855 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:10.855 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:28:10.855 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:28:10.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:28:10.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:28:10.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:28:11.219 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:11.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:28:11.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:28:11.644 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:28:11.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:28:21.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:28:21.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:28:21.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:28:22.134 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:28:22.135 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:28:22.135 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:28:22.135 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:22.135 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:28:22.136 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:28:22.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:28:22.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:28:22.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:28:22.525 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:22.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:28:22.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:28:22.908 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:28:22.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:28:32.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:28:32.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:28:33.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:28:33.400 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:28:33.400 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:28:33.400 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:28:33.401 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:33.401 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:28:33.401 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:28:33.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:28:33.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:28:33.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:28:33.767 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:33.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:28:33.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:28:34.234 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:28:34.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:28:44.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:28:44.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:28:44.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:28:44.732 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:28:44.733 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:28:44.733 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:28:44.733 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:44.733 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:28:44.734 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:28:44.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:28:44.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:28:44.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:28:45.101 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:45.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:28:45.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:28:45.481 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:28:45.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:28:55.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2021-05-12T12:29:10.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:29:10.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2021-05-12T12:29:10.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2021-05-12T12:29:10.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2021-05-12T12:29:10.983 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:29:10.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T12:29:10.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:29:10.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:29:10.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:29:10.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:29:10.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:29:10.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:29:10.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:29:10.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:29:11.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:29:11.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:29:11.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:29:11.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:29:11.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:29:11.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:29:11.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:29:11.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:29:11.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:29:11.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:29:11.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:29:11.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:29:11.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:29:11.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:29:11.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:29:11.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:11.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:11.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169344 2021-05-12T12:29:11.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:29:11.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:29:11.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T12:29:11.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T12:29:11.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T12:29:11.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T12:29:11.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:29:11.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:29:11.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:29:11.108 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:29:11.108 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:29:11.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:29:11.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:29:11.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:29:11.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:29:11.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:29:11.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:29:11.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:29:11.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:29:11.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:29:11.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:29:11.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:29:11.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:29:11.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:29:11.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:29:11.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:29:11.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:29:11.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:29:11.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:29:11.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:11.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:11.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169344 2021-05-12T12:29:11.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:29:11.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:29:11.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T12:29:11.117 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:29:11.117 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:11.117 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:11.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.169344 2021-05-12T12:29:11.119 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:29:11.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:29:11.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:29:11.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T12:29:11.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2021-05-12T12:29:11.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2021-05-12T12:29:11.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2021-05-12T12:29:11.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T12:29:11.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:29:11.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:29:11.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:29:11.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T12:29:11.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T12:29:11.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:29:11.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:11.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:11.485 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:11.485 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:11.486 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:11.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:29:11.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:29:11.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:29:11.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:29:11.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:29:11.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:29:11.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:29:11.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:29:11.748 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:29:11.748 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:29:11.748 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:29:11.749 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:29:11.749 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:11.749 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:11.749 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169344/ceph-mon.a.asok 2021-05-12T12:29:11.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:29:11.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.169344/ceph-mon.a.asok config get fsid 2021-05-12T12:29:11.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:29:11.846 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:29:11.846 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:29:11.846 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:29:11.846 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:29:11.847 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:29:11.847 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:29:11.847 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:29:11.847 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:29:11.848 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:11.848 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:11.848 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169344/ceph-mon.a.asok 2021-05-12T12:29:11.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:29:11.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.169344/ceph-mon.a.asok config get mon_host 2021-05-12T12:29:11.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2021-05-12T12:29:11.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T12:29:11.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:29:11.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:29:11.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:29:11.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T12:29:11.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:29:12.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:29:12.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:12.808 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:12.808 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:12.808 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:12.808 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:12.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:29:12.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:29:12.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:29:12.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2021-05-12T12:29:12.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:29:12.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:29:12.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:29:12.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:29:12.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T12:29:12.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:29:12.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:29:12.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:29:12.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:29:12.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T12:29:12.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T12:29:12.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:29:12.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:29:12.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:29:12.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:29:12.837 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:12.837 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:12.837 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:12.837 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:12.838 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:12.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:29:12.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:29:12.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:29:12.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:29:12.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:29:12.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:29:12.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:29:12.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:29:12.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:29:12.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:12.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:29:12.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:29:12.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T12:29:12.841 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:29:12.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e6076b40-a999-4b17-ae4a-28328150f225 2021-05-12T12:29:12.841 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 e6076b40-a999-4b17-ae4a-28328150f225 2021-05-12T12:29:12.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 e6076b40-a999-4b17-ae4a-28328150f225' 2021-05-12T12:29:12.842 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:29:12.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAYyptg51w4MhAAdra/qEeObmWYD4xrJ3cTTg== 2021-05-12T12:29:12.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAYyptg51w4MhAAdra/qEeObmWYD4xrJ3cTTg=="}' 2021-05-12T12:29:12.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e6076b40-a999-4b17-ae4a-28328150f225 -i td/osd-markdown/0/new.json 2021-05-12T12:29:13.154 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:29:13.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T12:29:13.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAYyptg51w4MhAAdra/qEeObmWYD4xrJ3cTTg== --osd-uuid e6076b40-a999-4b17-ae4a-28328150f225 2021-05-12T12:29:13.187 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:13.184+0000 7f33dc71b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:13.189 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:13.186+0000 7f33dc71b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:13.190 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:13.187+0000 7f33dc71b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:13.191 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:13.187+0000 7f33dc71b0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:29:13.191 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:13.187+0000 7f33dc71b0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:29:13.191 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:13.187+0000 7f33dc71b0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:29:13.228 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:13.225+0000 7f33dc71b0c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T12:29:17.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T12:29:17.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:29:17.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:29:17.798 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:29:17.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:29:18.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:29:18.277 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:29:18.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:18.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:29:18.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:29:18.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:29:18.296 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:18.293+0000 7f2bb77300c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:18.298 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:18.295+0000 7f2bb77300c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:18.299 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:18.296+0000 7f2bb77300c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:18.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:29:18.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:29:18.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:29:18.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:29:18.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:29:18.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:18.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:29:18.637 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:29:18.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:18.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:29:18.816 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:18.813+0000 7f2bb77300c0 -1 Falling back to public interface 2021-05-12T12:29:18.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:19.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:19.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:19.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:29:19.997 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:29:19.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:19.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:29:20.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:21.002 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:20.999+0000 7f2bb77300c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:29:21.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:21.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:21.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:29:21.350 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:29:21.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:21.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:29:21.713 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/180431,v1:127.0.0.1:6803/180431] [v2:127.0.0.1:6804/180431,v1:127.0.0.1:6805/180431] exists,up e6076b40-a999-4b17-ae4a-28328150f225 2021-05-12T12:29:21.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:29:21.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:29:21.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:29:21.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2021-05-12T12:29:21.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:29:21.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:29:21.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:29:21.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:29:21.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T12:29:21.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:29:21.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:29:21.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:29:21.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:29:21.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T12:29:21.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T12:29:21.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:29:21.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:29:21.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:29:21.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:29:21.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:21.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:21.726 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:21.726 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:21.726 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:21.726 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:29:21.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:29:21.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:29:21.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:29:21.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:29:21.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:29:21.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:29:21.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:29:21.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:29:21.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:21.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:29:21.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:29:21.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T12:29:21.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:29:21.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=801772fd-6098-41db-919b-09178e7ab8e9 2021-05-12T12:29:21.730 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 801772fd-6098-41db-919b-09178e7ab8e9 2021-05-12T12:29:21.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 801772fd-6098-41db-919b-09178e7ab8e9' 2021-05-12T12:29:21.731 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:29:21.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAhyptg3VpDKxAA7WRA1ba26Gu8T0ucwRbj1Q== 2021-05-12T12:29:21.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAhyptg3VpDKxAA7WRA1ba26Gu8T0ucwRbj1Q=="}' 2021-05-12T12:29:21.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 801772fd-6098-41db-919b-09178e7ab8e9 -i td/osd-markdown/1/new.json 2021-05-12T12:29:22.258 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:29:22.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T12:29:22.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAhyptg3VpDKxAA7WRA1ba26Gu8T0ucwRbj1Q== --osd-uuid 801772fd-6098-41db-919b-09178e7ab8e9 2021-05-12T12:29:22.291 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:22.288+0000 7f76b007b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:22.293 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:22.290+0000 7f76b007b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:22.294 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:22.291+0000 7f76b007b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:22.295 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:22.291+0000 7f76b007b0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:29:22.295 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:22.291+0000 7f76b007b0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:29:22.295 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:22.291+0000 7f76b007b0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:29:22.373 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:22.369+0000 7f76b007b0c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T12:29:27.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T12:29:27.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:29:27.124 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:29:27.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:29:27.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:29:27.623 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:29:27.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:29:27.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:27.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:29:27.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:29:27.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:29:27.642 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:27.639+0000 7f4af08dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:27.644 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:27.641+0000 7f4af08dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:27.646 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:27.642+0000 7f4af08dd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:27.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:29:27.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:29:27.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:29:27.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:29:27.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:29:27.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:27.982 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:29:27.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:29:27.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:27.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:29:28.163 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:28.159+0000 7f4af08dd0c0 -1 Falling back to public interface 2021-05-12T12:29:28.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:29.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:29.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:29.342 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:29:29.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:29:29.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:29.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:29:29.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:30.333 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:30.330+0000 7f4af08dd0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:29:30.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:30.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:30.717 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:29:30.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:29:30.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:30.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:29:31.075 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/181824,v1:127.0.0.1:6811/181824] [v2:127.0.0.1:6812/181824,v1:127.0.0.1:6813/181824] exists,up 801772fd-6098-41db-919b-09178e7ab8e9 2021-05-12T12:29:31.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:29:31.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:29:31.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:29:31.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2021-05-12T12:29:31.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:29:31.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:29:31.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:29:31.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:29:31.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T12:29:31.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:29:31.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:29:31.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:29:31.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:29:31.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T12:29:31.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T12:29:31.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:29:31.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:29:31.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:29:31.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:29:31.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:31.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:31.084 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:31.085 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:31.085 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:29:31.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:29:31.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:29:31.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:29:31.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:29:31.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:29:31.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:29:31.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:29:31.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:29:31.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:29:31.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:31.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:29:31.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:29:31.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T12:29:31.088 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:29:31.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=4215b529-6323-46c3-b496-20055183a715 2021-05-12T12:29:31.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 4215b529-6323-46c3-b496-20055183a715' 2021-05-12T12:29:31.089 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 4215b529-6323-46c3-b496-20055183a715 2021-05-12T12:29:31.089 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:29:31.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAryptgn/NIBRAAj+kQeltZ1TZinbadFgH2zw== 2021-05-12T12:29:31.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAryptgn/NIBRAAj+kQeltZ1TZinbadFgH2zw=="}' 2021-05-12T12:29:31.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 4215b529-6323-46c3-b496-20055183a715 -i td/osd-markdown/2/new.json 2021-05-12T12:29:31.616 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:29:31.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T12:29:31.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAryptgn/NIBRAAj+kQeltZ1TZinbadFgH2zw== --osd-uuid 4215b529-6323-46c3-b496-20055183a715 2021-05-12T12:29:31.649 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:31.646+0000 7f23fbd410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:31.666 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:31.663+0000 7f23fbd410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:31.667 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:31.664+0000 7f23fbd410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:31.668 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:31.665+0000 7f23fbd410c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:29:31.668 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:31.665+0000 7f23fbd410c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:29:31.668 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:31.665+0000 7f23fbd410c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:29:31.733 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:31.729+0000 7f23fbd410c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T12:29:36.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T12:29:36.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:29:36.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:29:36.299 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:29:36.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:29:36.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:29:36.781 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:29:36.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:36.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:29:36.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:29:36.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:29:36.800 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:36.796+0000 7fb3d3f190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:36.802 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:36.799+0000 7fb3d3f190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:36.803 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:36.800+0000 7fb3d3f190c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:37.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:29:37.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:29:37.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:29:37.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:29:37.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:29:37.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:37.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:29:37.145 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:29:37.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:37.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:29:37.327 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:37.324+0000 7fb3d3f190c0 -1 Falling back to public interface 2021-05-12T12:29:37.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:38.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:38.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:38.520 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:29:38.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:29:38.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:38.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:29:38.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:39.559 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:29:39.556+0000 7fb3d3f190c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:29:39.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:39.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:39.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:29:39.883 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:29:39.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:39.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:29:40.248 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/183209,v1:127.0.0.1:6819/183209] [v2:127.0.0.1:6820/183209,v1:127.0.0.1:6821/183209] exists,up 4215b529-6323-46c3-b496-20055183a715 2021-05-12T12:29:40.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:29:40.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:29:40.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:29:40.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2021-05-12T12:29:40.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:29:40.597 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' does not exist 2021-05-12T12:29:40.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:29:40.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:29:41.406 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'rbd' already exists 2021-05-12T12:29:41.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:29:42.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:29:43.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2021-05-12T12:29:43.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2021-05-12T12:29:43.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2021-05-12T12:29:43.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-05-12T12:29:43.657 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:29:43.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2021-05-12T12:29:43.784 INFO:tasks.workunit.client.0.smithi191.stdout:{} 2021-05-12T12:29:43.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2021-05-12T12:29:43.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-05-12T12:29:43.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2021-05-12T12:29:43.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-05-12T12:29:43.795 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-05-12T12:29:43.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:29:43.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:29:43.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:29:44.270 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:29:44.271 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:29:44.271 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:29:44.271 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:29:44.271 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:29:44.272 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:29:44.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:29:44.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:29:44.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:29:44.636 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:29:44.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:29:44.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:29:45.014 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:29:45.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:29:55.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:29:55.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:29:55.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:29:55.505 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:29:55.505 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:29:55.505 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:29:55.506 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:29:55.506 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:29:55.506 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:29:55.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:29:55.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:29:55.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:29:55.873 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:29:55.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:29:55.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:29:56.268 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:29:56.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:30:06.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:30:06.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:30:06.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:30:06.758 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:30:06.758 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:30:06.758 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:30:06.759 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:06.759 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:30:06.759 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:30:06.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:30:06.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:30:06.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:30:07.151 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:07.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:30:07.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:30:07.676 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:30:07.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:30:17.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:30:17.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:30:17.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:30:18.184 INFO:tasks.workunit.client.0.smithi191.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:30:18.184 INFO:tasks.workunit.client.0.smithi191.stdout:-1 0.29306 root default 2021-05-12T12:30:18.184 INFO:tasks.workunit.client.0.smithi191.stdout:-3 0.29306 host smithi191 2021-05-12T12:30:18.184 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:18.185 INFO:tasks.workunit.client.0.smithi191.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:30:18.185 INFO:tasks.workunit.client.0.smithi191.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:30:18.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:30:18.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:30:18.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:30:18.550 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:18.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:30:18.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:30:18.958 INFO:tasks.workunit.client.0.smithi191.stderr:marked down osd.0. 2021-05-12T12:30:18.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:30:28.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2021-05-12T12:30:28.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2021-05-12T12:30:28.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2021-05-12T12:30:29.336 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-05-12T12:30:29.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T12:30:29.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:30:29.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:30:29.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:30:29.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:30:29.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:30:29.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:30:29.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:30:29.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:30:29.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:30:29.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:30:29.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:30:29.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:30:29.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:30:29.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:30:29.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:30:29.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:30:29.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:30:29.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:30:29.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:30:29.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:30:29.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:30:29.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:30:29.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:30:29.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:29.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:29.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169344 2021-05-12T12:30:29.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:30:29.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:30:29.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T12:30:29.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T12:30:29.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T12:30:29.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T12:30:29.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:30:29.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:30:29.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:30:29.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:30:29.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:30:29.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:30:29.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:30:29.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:30:29.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:30:29.462 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:30:29.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:30:29.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:30:29.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:30:29.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:30:29.464 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:30:29.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:30:29.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:30:29.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:30:29.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:30:29.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:30:29.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:30:29.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:30:29.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:30:29.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:29.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:29.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169344 2021-05-12T12:30:29.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:30:29.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:30:29.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T12:30:29.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:30:29.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:29.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:29.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.169344 2021-05-12T12:30:29.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:30:29.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:30:29.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:30:29.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T12:30:29.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2021-05-12T12:30:29.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2021-05-12T12:30:29.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2021-05-12T12:30:29.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T12:30:29.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:30:29.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:30:29.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:30:29.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T12:30:29.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T12:30:29.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:30:29.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:29.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:29.824 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:29.824 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:29.824 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:29.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:30:29.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:30:30.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:30:30.051 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:30:30.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:30:30.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:30:30.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:30:30.053 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:30:30.053 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:30:30.053 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:30:30.053 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:30:30.054 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:30:30.054 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:30.054 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:30.054 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169344/ceph-mon.a.asok 2021-05-12T12:30:30.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:30:30.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.169344/ceph-mon.a.asok config get fsid 2021-05-12T12:30:30.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:30:30.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:30:30.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:30:30.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:30:30.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:30:30.148 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:30:30.148 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:30:30.148 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:30:30.149 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:30:30.149 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:30.149 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:30.149 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.169344/ceph-mon.a.asok 2021-05-12T12:30:30.150 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:30:30.150 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.169344/ceph-mon.a.asok config get mon_host 2021-05-12T12:30:30.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2021-05-12T12:30:30.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T12:30:30.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:30:30.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:30:30.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:30:30.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T12:30:30.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:30:31.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:30:31.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:31.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:31.105 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:31.105 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:31.106 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:31.106 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:30:31.106 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:30:31.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:30:31.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2021-05-12T12:30:31.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:30:31.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:30:31.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:30:31.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:30:31.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T12:30:31.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:30:31.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:30:31.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:30:31.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:30:31.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T12:30:31.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T12:30:31.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:30:31.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:30:31.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:30:31.133 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:30:31.133 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:31.133 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:31.134 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:31.134 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:31.134 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:31.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:30:31.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:30:31.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:30:31.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:30:31.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:30:31.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:30:31.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:30:31.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:30:31.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:30:31.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:31.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:30:31.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:30:31.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T12:30:31.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:30:31.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=79bf305d-2821-4278-b432-eb0d58c6bc7b 2021-05-12T12:30:31.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 79bf305d-2821-4278-b432-eb0d58c6bc7b' 2021-05-12T12:30:31.138 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 79bf305d-2821-4278-b432-eb0d58c6bc7b 2021-05-12T12:30:31.139 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:30:31.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBnyptgL41VCBAAIOC34OtJkcFY8JDXcRiaUw== 2021-05-12T12:30:31.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBnyptgL41VCBAAIOC34OtJkcFY8JDXcRiaUw=="}' 2021-05-12T12:30:31.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 79bf305d-2821-4278-b432-eb0d58c6bc7b -i td/osd-markdown/0/new.json 2021-05-12T12:30:31.450 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:30:31.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T12:30:31.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBnyptgL41VCBAAIOC34OtJkcFY8JDXcRiaUw== --osd-uuid 79bf305d-2821-4278-b432-eb0d58c6bc7b 2021-05-12T12:30:31.483 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:31.479+0000 7f59a302a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:31.484 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:31.481+0000 7f59a302a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:31.486 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:31.482+0000 7f59a302a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:31.487 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:31.483+0000 7f59a302a0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:30:31.487 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:31.483+0000 7f59a302a0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:30:31.487 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:31.483+0000 7f59a302a0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:30:31.516 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:31.513+0000 7f59a302a0c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T12:30:36.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T12:30:36.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:30:36.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:30:36.071 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:30:36.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:30:36.542 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:30:36.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:30:36.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:36.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:30:36.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:30:36.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:30:36.562 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:36.559+0000 7f1ed5d590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:36.563 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:36.560+0000 7f1ed5d590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:36.564 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:36.561+0000 7f1ed5d590c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:36.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:30:36.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:30:36.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:30:36.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:30:36.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:30:36.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:36.903 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:30:36.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:30:36.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:36.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:30:37.086 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:37.083+0000 7f1ed5d590c0 -1 Falling back to public interface 2021-05-12T12:30:37.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:38.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:38.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:38.258 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:30:38.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:30:38.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:38.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:30:38.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:39.273 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:39.270+0000 7f1ed5d590c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:30:39.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:39.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:39.619 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:30:39.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:30:39.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:39.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:30:39.976 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/185447,v1:127.0.0.1:6803/185447] [v2:127.0.0.1:6804/185447,v1:127.0.0.1:6805/185447] exists,up 79bf305d-2821-4278-b432-eb0d58c6bc7b 2021-05-12T12:30:39.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:30:39.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:30:39.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:30:39.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2021-05-12T12:30:39.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:30:39.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:30:39.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:30:39.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:30:39.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T12:30:39.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:30:39.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:30:39.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:30:39.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:30:39.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T12:30:39.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T12:30:39.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:30:39.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:30:39.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:30:39.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:30:39.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:39.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:39.986 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:39.986 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:39.986 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:39.986 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:30:39.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:30:39.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:30:39.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:30:39.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:30:39.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:30:39.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:30:39.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:30:39.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:30:39.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:39.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:30:39.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:30:39.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T12:30:39.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:30:39.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=8e22e5f1-74a0-406f-b497-61b6b063471a 2021-05-12T12:30:39.991 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 8e22e5f1-74a0-406f-b497-61b6b063471a 2021-05-12T12:30:39.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 8e22e5f1-74a0-406f-b497-61b6b063471a' 2021-05-12T12:30:39.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:30:39.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBvyptg1yv0OhAAV/9qf+rTQE0eyJMk8gtYtw== 2021-05-12T12:30:39.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBvyptg1yv0OhAAV/9qf+rTQE0eyJMk8gtYtw=="}' 2021-05-12T12:30:39.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 8e22e5f1-74a0-406f-b497-61b6b063471a -i td/osd-markdown/1/new.json 2021-05-12T12:30:40.496 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:30:40.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T12:30:40.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBvyptg1yv0OhAAV/9qf+rTQE0eyJMk8gtYtw== --osd-uuid 8e22e5f1-74a0-406f-b497-61b6b063471a 2021-05-12T12:30:40.529 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:40.526+0000 7f426e6d30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:40.554 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:40.551+0000 7f426e6d30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:40.556 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:40.552+0000 7f426e6d30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:40.556 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:40.553+0000 7f426e6d30c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:30:40.557 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:40.553+0000 7f426e6d30c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:30:40.557 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:40.553+0000 7f426e6d30c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:30:40.617 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:40.614+0000 7f426e6d30c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T12:30:45.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T12:30:45.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:30:45.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:30:45.385 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:30:45.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:30:45.868 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:30:45.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:30:45.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:30:45.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:45.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:30:45.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:30:45.887 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:45.883+0000 7f29244800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:45.888 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:45.885+0000 7f29244800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:45.889 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:45.886+0000 7f29244800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:46.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:30:46.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:30:46.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:30:46.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:30:46.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:30:46.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:46.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:30:46.231 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:30:46.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:46.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:30:46.405 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:46.401+0000 7f29244800c0 -1 Falling back to public interface 2021-05-12T12:30:46.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:47.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:47.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:47.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:30:47.604 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:30:47.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:47.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:30:47.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:48.593 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:48.589+0000 7f29244800c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:30:48.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:48.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:48.972 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:30:48.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:30:48.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:48.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:30:49.339 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/186832,v1:127.0.0.1:6811/186832] [v2:127.0.0.1:6812/186832,v1:127.0.0.1:6813/186832] exists,up 8e22e5f1-74a0-406f-b497-61b6b063471a 2021-05-12T12:30:49.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:30:49.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:30:49.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:30:49.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2021-05-12T12:30:49.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:30:49.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:30:49.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:30:49.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:30:49.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T12:30:49.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:30:49.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:30:49.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:30:49.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:30:49.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T12:30:49.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T12:30:49.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:30:49.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:30:49.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:30:49.348 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:30:49.348 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:49.348 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:49.349 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:49.349 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:49.349 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:30:49.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:30:49.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' 2021-05-12T12:30:49.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:30:49.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:30:49.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:30:49.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:30:49.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:30:49.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:30:49.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:30:49.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:49.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:30:49.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:30:49.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T12:30:49.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:30:49.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=4d528123-012d-475a-af9e-3e6b2e3d9a90 2021-05-12T12:30:49.353 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 4d528123-012d-475a-af9e-3e6b2e3d9a90 2021-05-12T12:30:49.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 4d528123-012d-475a-af9e-3e6b2e3d9a90' 2021-05-12T12:30:49.354 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:30:49.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB5yptg8KXqFBAAtGzxbu01czK2zjjlosOIiA== 2021-05-12T12:30:49.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB5yptg8KXqFBAAtGzxbu01czK2zjjlosOIiA=="}' 2021-05-12T12:30:49.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 4d528123-012d-475a-af9e-3e6b2e3d9a90 -i td/osd-markdown/2/new.json 2021-05-12T12:30:49.897 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:30:49.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T12:30:49.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB5yptg8KXqFBAAtGzxbu01czK2zjjlosOIiA== --osd-uuid 4d528123-012d-475a-af9e-3e6b2e3d9a90 2021-05-12T12:30:49.930 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:49.927+0000 7f5a8431e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:49.932 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:49.929+0000 7f5a8431e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:49.933 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:49.929+0000 7f5a8431e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:49.933 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:49.930+0000 7f5a8431e0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:30:49.934 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:49.930+0000 7f5a8431e0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:30:49.934 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:49.930+0000 7f5a8431e0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:30:49.973 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:49.970+0000 7f5a8431e0c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T12:30:54.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T12:30:54.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:30:54.856 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:30:54.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:30:54.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:30:55.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:30:55.343 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:30:55.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=a439a5ac-cfd1-4285-9fde-b7e9ab6003cd --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.169344/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:55.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:30:55.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:30:55.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:30:55.362 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:55.358+0000 7f37dbc9a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:55.363 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:55.360+0000 7f37dbc9a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:55.364 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:55.361+0000 7f37dbc9a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:55.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:30:55.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:30:55.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:30:55.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:30:55.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:30:55.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:55.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:30:55.704 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:30:55.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:55.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:30:55.880 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:55.877+0000 7f37dbc9a0c0 -1 Falling back to public interface 2021-05-12T12:30:56.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:57.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:57.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:57.059 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:30:57.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:30:57.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:57.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:30:57.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:58.075 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:30:58.072+0000 7f37dbc9a0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:30:58.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:58.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:58.432 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:30:58.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:30:58.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:58.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:30:58.799 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/188217,v1:127.0.0.1:6819/188217] [v2:127.0.0.1:6820/188217,v1:127.0.0.1:6821/188217] exists,up 4d528123-012d-475a-af9e-3e6b2e3d9a90 2021-05-12T12:30:58.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:30:58.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:30:58.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:30:58.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2021-05-12T12:30:58.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=185447 2021-05-12T12:30:58.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 185447 2021-05-12T12:30:58.806 INFO:tasks.workunit.client.0.smithi191.stdout: PID TTY TIME CMD 2021-05-12T12:30:58.806 INFO:tasks.workunit.client.0.smithi191.stdout: 185447 ? 00:00:00 ceph-osd 2021-05-12T12:30:58.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2021-05-12T12:30:58.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2021-05-12T12:30:58.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2021-05-12T12:30:59.163 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:59.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2021-05-12T12:30:59.720 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already stopped. 2021-05-12T12:30:59.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2021-05-12T12:31:14.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2021-05-12T12:31:14.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2021-05-12T12:31:14.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2021-05-12T12:31:15.095 INFO:tasks.workunit.client.0.smithi191.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-05-12T12:31:15.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 185447 2021-05-12T12:31:15.099 INFO:tasks.workunit.client.0.smithi191.stdout: PID TTY TIME CMD 2021-05-12T12:31:15.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T12:31:15.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:31:15.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:31:15.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:31:15.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:15.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:15.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:15.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:15.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:15.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:15.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:15.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:15.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:15.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:15.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:15.211 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:15.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:15.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:15.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:15.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:15.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:15.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:31:15.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:31:15.218 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:15.218 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:15.218 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:31:15.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169344 2021-05-12T12:31:15.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:15.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:15.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:31:15.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-markdown 0 2021-05-12T12:31:15.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:31:15.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:31:15.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:31:15.222 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:15.222 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:15.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:15.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:15.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:15.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:15.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:15.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:15.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:15.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:15.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:15.226 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:15.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:15.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:15.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:15.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:15.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:15.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:31:15.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:31:15.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:15.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:15.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.169344 2021-05-12T12:31:15.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.169344 2021-05-12T12:31:15.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:15.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:15.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:31:15.233 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:31:15.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:31:15.265 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2021-05-12T12:31:15.265 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2021-05-12T12:31:15.325 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:31:15.335 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:31:15.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:31:15.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:31:15.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:31:15.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:31:15.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:31:15.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:31:15.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:31:15.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:31:15.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-reactivate 2021-05-12T12:31:15.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2021-05-12T12:31:15.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2021-05-12T12:31:15.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2021-05-12T12:31:15.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2021-05-12T12:31:15.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2021-05-12T12:31:15.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2021-05-12T12:31:15.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=63f0eaec-92de-40d9-acb6-9f20a157691b --auth-supported=none ' 2021-05-12T12:31:15.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2021-05-12T12:31:15.339 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2021-05-12T12:31:15.340 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:31:15.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2021-05-12T12:31:15.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2021-05-12T12:31:15.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2021-05-12T12:31:15.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2021-05-12T12:31:15.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2021-05-12T12:31:15.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-05-12T12:31:15.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:31:15.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-05-12T12:31:15.342 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:15.342 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:15.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:15.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:15.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:15.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:15.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:15.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:15.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:15.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:15.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:15.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:15.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:15.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:15.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:15.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:15.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:15.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:31:15.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-05-12T12:31:15.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:15.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:15.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:15.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188922 2021-05-12T12:31:15.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:15.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:15.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2021-05-12T12:31:15.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:31:15.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:15.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:15.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188922 2021-05-12T12:31:15.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:31:15.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:31:15.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:31:15.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:31:15.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2021-05-12T12:31:15.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2021-05-12T12:31:15.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2021-05-12T12:31:15.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2021-05-12T12:31:15.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2021-05-12T12:31:15.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:31:15.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:31:15.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:31:15.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2021-05-12T12:31:15.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2021-05-12T12:31:15.699 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:31:15.699 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:15.699 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:15.699 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:15.701 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:15.701 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:15.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188922/$cluster-$name.asok' 2021-05-12T12:31:15.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.188922/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:31:15.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:31:15.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:31:15.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:31:15.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:31:15.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:31:15.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:31:15.928 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:31:15.928 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:31:15.928 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:31:15.929 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:31:15.929 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:15.929 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:15.929 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188922/ceph-mon.a.asok 2021-05-12T12:31:15.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:31:15.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.188922/ceph-mon.a.asok config get fsid 2021-05-12T12:31:16.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:31:16.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:31:16.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:31:16.022 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:31:16.022 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:31:16.022 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:31:16.023 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:31:16.023 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:31:16.023 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:31:16.023 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:16.024 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:16.024 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188922/ceph-mon.a.asok 2021-05-12T12:31:16.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:31:16.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.188922/ceph-mon.a.asok config get mon_host 2021-05-12T12:31:16.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2021-05-12T12:31:16.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2021-05-12T12:31:16.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:31:16.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:31:16.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:31:16.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2021-05-12T12:31:16.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:31:16.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:31:16.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:16.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:16.978 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:16.979 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:16.979 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:16.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188922/$cluster-$name.asok' 2021-05-12T12:31:16.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:31:16.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.188922/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:31:17.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2021-05-12T12:31:17.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2021-05-12T12:31:17.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:31:17.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:31:17.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:31:17.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2021-05-12T12:31:17.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=63f0eaec-92de-40d9-acb6-9f20a157691b --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-05-12T12:31:17.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:31:17.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:31:17.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:31:17.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-05-12T12:31:17.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-05-12T12:31:17.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:31:17.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:31:17.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-05-12T12:31:17.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:31:17.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:17.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:17.008 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:17.008 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:17.009 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:17.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188922/$cluster-$name.asok' 2021-05-12T12:31:17.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188922/$cluster-$name.asok' 2021-05-12T12:31:17.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:31:17.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:31:17.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:31:17.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-05-12T12:31:17.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-05-12T12:31:17.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:31:17.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:31:17.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:17.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:31:17.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:31:17.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-reactivate/0 2021-05-12T12:31:17.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:31:17.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=d94ca158-c026-4942-9ccc-ebe7aff402f0 2021-05-12T12:31:17.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 d94ca158-c026-4942-9ccc-ebe7aff402f0' 2021-05-12T12:31:17.013 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 d94ca158-c026-4942-9ccc-ebe7aff402f0 2021-05-12T12:31:17.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:31:17.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCVyptg+4rDABAAbplw+XO87bPG8AyyHp8XTQ== 2021-05-12T12:31:17.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCVyptg+4rDABAAbplw+XO87bPG8AyyHp8XTQ=="}' 2021-05-12T12:31:17.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new d94ca158-c026-4942-9ccc-ebe7aff402f0 -i td/osd-reactivate/0/new.json 2021-05-12T12:31:17.333 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:31:17.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-reactivate/0/new.json 2021-05-12T12:31:17.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=63f0eaec-92de-40d9-acb6-9f20a157691b --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.188922/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCVyptg+4rDABAAbplw+XO87bPG8AyyHp8XTQ== --osd-uuid d94ca158-c026-4942-9ccc-ebe7aff402f0 2021-05-12T12:31:17.365 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:17.362+0000 7f1236ceb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:17.368 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:17.364+0000 7f1236ceb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:17.368 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:17.365+0000 7f1236ceb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:17.369 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:17.366+0000 7f1236ceb0c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-05-12T12:31:17.369 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:17.366+0000 7f1236ceb0c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-05-12T12:31:17.369 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:17.366+0000 7f1236ceb0c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-05-12T12:31:17.407 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:17.403+0000 7f1236ceb0c0 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2021-05-12T12:31:21.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-reactivate/0/keyring 2021-05-12T12:31:21.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:31:21.729 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:31:21.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:31:21.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:31:22.199 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:31:22.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:31:22.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=63f0eaec-92de-40d9-acb6-9f20a157691b --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.188922/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:22.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:31:22.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:31:22.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:31:22.219 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:22.215+0000 7fc3b828d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:22.220 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:22.217+0000 7fc3b828d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:22.221 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:22.218+0000 7fc3b828d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:22.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:31:22.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:31:22.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:31:22.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:31:22.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:31:22.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:22.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:31:22.556 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:31:22.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:22.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:22.737 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:22.733+0000 7fc3b828d0c0 -1 Falling back to public interface 2021-05-12T12:31:22.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:23.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:23.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:23.912 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:31:23.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:31:23.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:23.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:24.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:24.923 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:24.920+0000 7fc3b828d0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:31:25.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:25.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:25.273 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:31:25.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:31:25.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:25.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:25.659 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/190042,v1:127.0.0.1:6803/190042] [v2:127.0.0.1:6804/190042,v1:127.0.0.1:6805/190042] exists,up d94ca158-c026-4942-9ccc-ebe7aff402f0 2021-05-12T12:31:25.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:31:25.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:31:25.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:31:25.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2021-05-12T12:31:25.663 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:25.663 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:25.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:25.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:25.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:25.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:25.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2021-05-12T12:31:25.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2021-05-12T12:31:25.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2021-05-12T12:31:25.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2021-05-12T12:31:25.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-reactivate 2021-05-12T12:31:25.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:31:25.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:31:25.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:31:25.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-reactivate/0 2021-05-12T12:31:25.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=63f0eaec-92de-40d9-acb6-9f20a157691b --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-05-12T12:31:25.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:31:25.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:31:25.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:31:25.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-05-12T12:31:25.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-05-12T12:31:25.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:31:25.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:31:25.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-05-12T12:31:25.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:31:25.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:25.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:25.770 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:25.770 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:25.770 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:25.771 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188922/$cluster-$name.asok' 2021-05-12T12:31:25.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188922/$cluster-$name.asok' 2021-05-12T12:31:25.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:31:25.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-05-12T12:31:25.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-05-12T12:31:25.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:31:25.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:31:25.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:25.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:31:25.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:31:25.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-reactivate/0 2021-05-12T12:31:25.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:31:25.774 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:31:25.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=63f0eaec-92de-40d9-acb6-9f20a157691b --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.188922/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:25.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-reactivate/0/whoami 2021-05-12T12:31:25.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:31:25.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:31:25.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:31:25.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:31:25.783 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:25.780+0000 7fd7104980c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:25.793 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:25.789+0000 7fd7104980c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:25.794 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:25.791+0000 7fd7104980c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:26.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:31:26.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:31:26.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:31:26.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:31:26.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:31:26.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:26.130 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:31:26.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:31:26.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:26.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:26.310 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:26.307+0000 7fd7104980c0 -1 Falling back to public interface 2021-05-12T12:31:26.507 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/190042,v1:127.0.0.1:6803/190042] [v2:127.0.0.1:6804/190042,v1:127.0.0.1:6805/190042] exists,up d94ca158-c026-4942-9ccc-ebe7aff402f0 2021-05-12T12:31:26.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:31:26.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:31:26.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:31:26.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2021-05-12T12:31:26.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-05-12T12:31:26.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:31:26.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-05-12T12:31:26.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:26.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:26.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:26.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:26.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:26.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:26.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:26.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:26.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:26.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:26.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:26.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:26.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:26.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:26.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:26.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:26.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:26.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:31:26.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-05-12T12:31:26.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:26.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:26.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:26.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188922 2021-05-12T12:31:26.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:26.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:26.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:31:26.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-reactivate 0 2021-05-12T12:31:26.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-05-12T12:31:26.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:31:26.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-05-12T12:31:26.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:26.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:26.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:26.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:26.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:26.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:26.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:26.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:26.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:26.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:26.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:26.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:26.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:26.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:26.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:26.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:26.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:26.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:31:26.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-05-12T12:31:26.633 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:26.633 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:26.633 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188922 2021-05-12T12:31:26.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188922 2021-05-12T12:31:26.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:26.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:26.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:31:26.634 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:31:26.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:31:26.697 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2021-05-12T12:31:26.698 DEBUG:teuthology.orchestra.run.smithi191:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2021-05-12T12:31:26.761 INFO:tasks.workunit.client.0.smithi191.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:31:26.768 INFO:tasks.workunit.client.0.smithi191.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:31:26.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:31:26.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:31:26.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:31:26.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:31:26.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:31:26.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:31:26.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:31:26.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:31:26.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-recovery-prio 2021-05-12T12:31:26.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2021-05-12T12:31:26.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2021-05-12T12:31:26.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2021-05-12T12:31:26.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2021-05-12T12:31:26.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2021-05-12T12:31:26.773 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2021-05-12T12:31:26.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none ' 2021-05-12T12:31:26.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:31:26.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: export objects=200 2021-05-12T12:31:26.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: objects=200 2021-05-12T12:31:26.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: export poolprefix=test 2021-05-12T12:31:26.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: poolprefix=test 2021-05-12T12:31:26.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: export FORCE_PRIO=255 2021-05-12T12:31:26.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: FORCE_PRIO=255 2021-05-12T12:31:26.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export NORMAL_PRIO=190 2021-05-12T12:31:26.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: NORMAL_PRIO=190 2021-05-12T12:31:26.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: set 2021-05-12T12:31:26.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:31:26.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: local 'funcs=TEST_recovery_pool_priority 2021-05-12T12:31:26.777 INFO:tasks.workunit.client.0.smithi191.stderr:TEST_recovery_priority' 2021-05-12T12:31:26.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2021-05-12T12:31:26.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2021-05-12T12:31:26.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2021-05-12T12:31:26.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2021-05-12T12:31:26.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-05-12T12:31:26.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:31:26.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-05-12T12:31:26.778 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:26.779 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:26.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:26.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:26.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:26.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:26.785 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:26.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:26.786 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:26.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:26.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:26.787 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:26.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:26.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:26.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:26.787 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:26.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:26.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:31:26.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-05-12T12:31:26.788 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:26.788 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:26.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:31:26.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.190757 2021-05-12T12:31:26.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:26.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:26.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2021-05-12T12:31:26.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:31:26.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:26.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:31:26.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.190757 2021-05-12T12:31:26.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:31:26.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:31:26.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:31:26.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:31:26.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2021-05-12T12:31:26.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_pool_priority td/osd-recovery-prio 2021-05-12T12:31:26.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:323: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2021-05-12T12:31:26.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:324: TEST_recovery_pool_priority: local pools=3 2021-05-12T12:31:26.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:325: TEST_recovery_pool_priority: local OSDS=2 2021-05-12T12:31:26.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2021-05-12T12:31:26.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2021-05-12T12:31:26.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:31:26.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:31:26.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:31:26.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2021-05-12T12:31:26.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2021-05-12T12:31:27.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:31:27.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:27.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:27.135 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:27.135 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:27.135 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:31:27.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:31:27.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:31:27.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:31:27.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:31:27.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:31:27.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:31:27.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:31:27.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:31:27.388 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:31:27.388 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:31:27.388 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:31:27.389 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:31:27.389 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:27.392 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:31:27.392 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-mon.a.asok 2021-05-12T12:31:27.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:31:27.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.190757/ceph-mon.a.asok config get fsid 2021-05-12T12:31:27.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:31:27.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:31:27.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:31:27.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:31:27.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:31:27.482 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:31:27.483 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:31:27.483 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:31:27.483 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:31:27.483 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:27.483 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:31:27.484 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-mon.a.asok 2021-05-12T12:31:27.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:31:27.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.190757/ceph-mon.a.asok config get mon_host 2021-05-12T12:31:27.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2021-05-12T12:31:27.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2021-05-12T12:31:27.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:31:27.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:31:27.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:31:27.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2021-05-12T12:31:27.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:31:28.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:31:28.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:28.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:28.447 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:28.448 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:28.448 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:31:28.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:31:28.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:31:28.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:31:28.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:329: TEST_recovery_pool_priority: export CEPH_ARGS 2021-05-12T12:31:28.470 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: expr 2 - 1 2021-05-12T12:31:28.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: seq 0 1 2021-05-12T12:31:28.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:31:28.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2021-05-12T12:31:28.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:31:28.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:31:28.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:31:28.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:31:28.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2021-05-12T12:31:28.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:31:28.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:31:28.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:31:28.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:31:28.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2021-05-12T12:31:28.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2021-05-12T12:31:28.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:31:28.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:31:28.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:31:28.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:31:28.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:28.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:28.482 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:28.482 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:28.482 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:31:28.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:31:28.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:31:28.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:31:28.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:31:28.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:31:28.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:31:28.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:31:28.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:31:28.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:31:28.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:28.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:31:28.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:31:28.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/0 2021-05-12T12:31:28.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:31:28.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=afeedea2-3fd2-4caa-946f-b1485e0404a9 2021-05-12T12:31:28.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 afeedea2-3fd2-4caa-946f-b1485e0404a9' 2021-05-12T12:31:28.486 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 afeedea2-3fd2-4caa-946f-b1485e0404a9 2021-05-12T12:31:28.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:31:28.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCgyptg3tsFHRAAfhmCc2ya8cGhy5XMb/oTtQ== 2021-05-12T12:31:28.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCgyptg3tsFHRAAfhmCc2ya8cGhy5XMb/oTtQ=="}' 2021-05-12T12:31:28.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new afeedea2-3fd2-4caa-946f-b1485e0404a9 -i td/osd-recovery-prio/0/new.json 2021-05-12T12:31:28.818 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:31:28.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/0/new.json 2021-05-12T12:31:28.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCgyptg3tsFHRAAfhmCc2ya8cGhy5XMb/oTtQ== --osd-uuid afeedea2-3fd2-4caa-946f-b1485e0404a9 2021-05-12T12:31:28.853 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:28.850+0000 7f1943bfe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:28.855 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:28.852+0000 7f1943bfe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:28.857 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:28.853+0000 7f1943bfe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:28.858 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:28.855+0000 7f1943bfe0c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:31:28.858 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:28.855+0000 7f1943bfe0c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:31:28.858 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:28.855+0000 7f1943bfe0c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:31:28.892 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:28.888+0000 7f1943bfe0c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2021-05-12T12:31:33.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2021-05-12T12:31:33.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:31:33.450 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:31:33.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:31:33.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:31:33.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:31:33.930 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:31:33.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:33.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:31:33.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:31:33.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:31:33.952 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:33.948+0000 7fcd588080c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:33.953 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:33.950+0000 7fcd588080c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:33.954 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:33.951+0000 7fcd588080c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:34.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:31:34.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:31:34.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:31:34.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:31:34.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:31:34.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:34.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:31:34.293 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:31:34.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:34.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:34.470 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:34.466+0000 7fcd588080c0 -1 Falling back to public interface 2021-05-12T12:31:34.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:35.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:35.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:35.649 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:31:35.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:31:35.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:35.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:36.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:36.634 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:36.631+0000 7fcd588080c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:31:37.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:37.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:37.014 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:31:37.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:31:37.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:37.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:37.385 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/191880,v1:127.0.0.1:6803/191880] [v2:127.0.0.1:6804/191880,v1:127.0.0.1:6805/191880] exists,up afeedea2-3fd2-4caa-946f-b1485e0404a9 2021-05-12T12:31:37.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:31:37.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:31:37.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:31:37.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:31:37.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2021-05-12T12:31:37.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:31:37.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:31:37.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:31:37.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:31:37.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2021-05-12T12:31:37.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:31:37.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:31:37.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:31:37.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:31:37.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2021-05-12T12:31:37.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2021-05-12T12:31:37.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:31:37.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:31:37.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:31:37.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:31:37.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:37.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:37.403 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:37.404 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:37.406 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:31:37.406 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:31:37.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:31:37.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:31:37.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:31:37.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:31:37.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:31:37.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:31:37.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:31:37.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:31:37.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:37.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:31:37.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:31:37.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/1 2021-05-12T12:31:37.409 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:31:37.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=72b17655-9d4f-4706-a5d5-d64f5c1efd98 2021-05-12T12:31:37.411 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 72b17655-9d4f-4706-a5d5-d64f5c1efd98 2021-05-12T12:31:37.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 72b17655-9d4f-4706-a5d5-d64f5c1efd98' 2021-05-12T12:31:37.413 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:31:37.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCpyptgiLrKFxAATI9qWBLLpdWkTMvy8K2FXg== 2021-05-12T12:31:37.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCpyptgiLrKFxAATI9qWBLLpdWkTMvy8K2FXg=="}' 2021-05-12T12:31:37.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 72b17655-9d4f-4706-a5d5-d64f5c1efd98 -i td/osd-recovery-prio/1/new.json 2021-05-12T12:31:37.918 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:31:37.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/1/new.json 2021-05-12T12:31:37.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCpyptgiLrKFxAATI9qWBLLpdWkTMvy8K2FXg== --osd-uuid 72b17655-9d4f-4706-a5d5-d64f5c1efd98 2021-05-12T12:31:37.954 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:37.950+0000 7f6a13bb30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:37.981 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:37.977+0000 7f6a13bb30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:37.983 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:37.978+0000 7f6a13bb30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:37.992 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:37.979+0000 7f6a13bb30c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:31:37.992 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:37.979+0000 7f6a13bb30c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:31:37.993 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:37.979+0000 7f6a13bb30c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:31:38.080 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:38.077+0000 7f6a13bb30c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2021-05-12T12:31:42.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2021-05-12T12:31:42.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:31:42.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:31:42.726 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:31:42.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:31:43.212 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:31:43.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:31:43.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:43.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:31:43.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:31:43.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:31:43.239 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:43.230+0000 7f307cb760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:43.239 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:43.232+0000 7f307cb760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:43.240 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:43.234+0000 7f307cb760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:43.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:31:43.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:31:43.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:31:43.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:31:43.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:31:43.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:43.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:31:43.577 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:31:43.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:43.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:31:43.758 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:43.754+0000 7f307cb760c0 -1 Falling back to public interface 2021-05-12T12:31:43.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:44.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:44.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:44.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:31:44.936 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:31:44.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:44.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:31:45.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:45.999 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:31:45.995+0000 7f307cb760c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:31:46.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:46.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:46.301 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:31:46.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:31:46.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:46.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:31:46.671 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/193265,v1:127.0.0.1:6811/193265] [v2:127.0.0.1:6812/193265,v1:127.0.0.1:6813/193265] exists,up 72b17655-9d4f-4706-a5d5-d64f5c1efd98 2021-05-12T12:31:46.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:31:46.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:31:46.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:31:46.678 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: seq 1 3 2021-05-12T12:31:46.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:31:46.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test1 1 1 2021-05-12T12:31:46.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-05-12T12:31:47.267 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test1' already exists 2021-05-12T12:31:47.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:31:48.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-05-12T12:31:48.951 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 size to 2 2021-05-12T12:31:48.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:31:48.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test2 1 1 2021-05-12T12:31:48.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2021-05-12T12:31:49.523 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test2' already exists 2021-05-12T12:31:49.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:31:50.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-05-12T12:31:51.207 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 2 size to 2 2021-05-12T12:31:51.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:31:51.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test3 1 1 2021-05-12T12:31:51.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2021-05-12T12:31:51.755 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test3' already exists 2021-05-12T12:31:51.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:31:52.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2021-05-12T12:31:53.735 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 3 size to 2 2021-05-12T12:31:53.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: sleep 5 2021-05-12T12:31:58.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:343: TEST_recovery_pool_priority: wait_for_clean 2021-05-12T12:31:58.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:31:58.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:31:58.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:31:58.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:31:58.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:31:58.749 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:31:58.749 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:31:58.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:31:58.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:31:58.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:31:58.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:31:58.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:31:58.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:31:58.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:31:58.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:31:59.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:31:59.213 INFO:tasks.workunit.client.0.smithi191.stderr:1' 2021-05-12T12:31:59.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:31:59.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:31:59.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:31:59.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836487 2021-05-12T12:31:59.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836487 2021-05-12T12:31:59.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487' 2021-05-12T12:31:59.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:31:59.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:31:59.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672965 2021-05-12T12:31:59.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672965 2021-05-12T12:31:59.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-05-12T12:31:59.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:31:59.491 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-05-12T12:31:59.491 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:31:59.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:31:59.492 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836487 2021-05-12T12:31:59.493 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:31:59.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836487 2021-05-12T12:31:59.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-05-12T12:31:59.494 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836487 2021-05-12T12:31:59.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:31:59.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836485 -lt 21474836487 2021-05-12T12:31:59.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:32:00.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:32:00.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:32:01.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836487 -lt 21474836487 2021-05-12T12:32:01.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:32:01.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-05-12T12:32:01.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:32:01.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:32:01.226 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672965 2021-05-12T12:32:01.226 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:32:01.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672965 2021-05-12T12:32:01.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-05-12T12:32:01.229 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672965 2021-05-12T12:32:01.230 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:32:01.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672965 -lt 42949672965 2021-05-12T12:32:01.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:32:01.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:32:01.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:32:02.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 3 == 0 2021-05-12T12:32:02.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:32:02.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:32:02.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:32:02.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:32:02.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:32:02.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:32:02.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:32:02.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=3 2021-05-12T12:32:02.428 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:32:02.428 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:32:02.428 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:32:02.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 3 = 3 2021-05-12T12:32:02.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:32:02.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:32:02.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:345: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T12:32:03.235 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:32:03.236 INFO:tasks.workunit.client.0.smithi191.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:31:53.701411+0000 0'0 26:17 [1,0] 1 [1,0] 1 0'0 2021-05-12T12:31:51.541975+0000 0'0 2021-05-12T12:31:51.541975+0000 0 2021-05-12T12:32:03.236 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:31:49.199005+0000 0'0 26:27 [1,0] 1 [1,0] 1 0'0 2021-05-12T12:31:46.974016+0000 0'0 2021-05-12T12:31:46.974016+0000 0 2021-05-12T12:32:03.236 INFO:tasks.workunit.client.0.smithi191.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:31:52.272962+0000 0'0 26:23 [0,1] 0 [0,1] 0 0'0 2021-05-12T12:31:49.263826+0000 0'0 2021-05-12T12:31:49.263826+0000 0 2021-05-12T12:32:03.237 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:32:03.237 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:32:03.237 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:32:03.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:349: TEST_recovery_pool_priority: local PG1 2021-05-12T12:32:03.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:350: TEST_recovery_pool_priority: local POOLNUM1 2021-05-12T12:32:03.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:351: TEST_recovery_pool_priority: local pool1 2021-05-12T12:32:03.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local chk_osd1_1 2021-05-12T12:32:03.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local chk_osd1_2 2021-05-12T12:32:03.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local PG2 2021-05-12T12:32:03.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local POOLNUM2 2021-05-12T12:32:03.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:357: TEST_recovery_pool_priority: local pool2 2021-05-12T12:32:03.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local chk_osd2_1 2021-05-12T12:32:03.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local chk_osd2_2 2021-05-12T12:32:03.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: seq 1 3 2021-05-12T12:32:03.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:03.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2021-05-12T12:32:03.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-05-12T12:32:03.601 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:32:03.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=1 2021-05-12T12:32:03.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:32:03.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=0 2021-05-12T12:32:03.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z '' ']' 2021-05-12T12:32:03.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: PG1=1.0 2021-05-12T12:32:03.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: POOLNUM1=1 2021-05-12T12:32:03.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:370: TEST_recovery_pool_priority: pool1=test1 2021-05-12T12:32:03.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: chk_osd1_1=1 2021-05-12T12:32:03.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: chk_osd1_2=0 2021-05-12T12:32:03.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:03.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2021-05-12T12:32:03.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-05-12T12:32:03.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:32:03.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=0 2021-05-12T12:32:03.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:32:03.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=1 2021-05-12T12:32:03.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2021-05-12T12:32:03.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2021-05-12T12:32:03.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: PG2=2.0 2021-05-12T12:32:03.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: POOLNUM2=2 2021-05-12T12:32:03.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:377: TEST_recovery_pool_priority: pool2=test2 2021-05-12T12:32:03.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: chk_osd2_1=0 2021-05-12T12:32:03.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: chk_osd2_2=1 2021-05-12T12:32:03.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: break 2021-05-12T12:32:03.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2021-05-12T12:32:03.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:385: TEST_recovery_pool_priority: '[' test2 = '' ']' 2021-05-12T12:32:03.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: seq 1 3 2021-05-12T12:32:03.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:03.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2021-05-12T12:32:03.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:03.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2021-05-12T12:32:03.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:03.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2021-05-12T12:32:03.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:395: TEST_recovery_pool_priority: delete_pool test3 2021-05-12T12:32:03.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2021-05-12T12:32:03.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-05-12T12:32:04.472 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test3' does not exist 2021-05-12T12:32:04.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:399: TEST_recovery_pool_priority: pool1_extra_prio=1 2021-05-12T12:32:04.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:400: TEST_recovery_pool_priority: pool2_extra_prio=2 2021-05-12T12:32:04.505 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: expr 190 + 1 2021-05-12T12:32:04.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: pool1_prio=191 2021-05-12T12:32:04.506 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: expr 190 + 2 2021-05-12T12:32:04.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool2_prio=192 2021-05-12T12:32:04.516 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-05-12T12:32:05.353 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 size to 1 2021-05-12T12:32:05.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2021-05-12T12:32:06.091 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 recovery_priority to 1 2021-05-12T12:32:06.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:406: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2021-05-12T12:32:06.765 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 2 size to 1 2021-05-12T12:32:06.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2021-05-12T12:32:07.623 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 2 recovery_priority to 2 2021-05-12T12:32:07.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: wait_for_clean 2021-05-12T12:32:07.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:32:07.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:32:07.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:32:07.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:32:07.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:32:07.648 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:32:07.648 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:32:07.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:32:07.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:32:07.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:32:07.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:32:07.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:32:07.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:32:07.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:32:07.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:32:08.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:32:08.078 INFO:tasks.workunit.client.0.smithi191.stderr:1' 2021-05-12T12:32:08.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:32:08.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:32:08.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:32:08.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836491 2021-05-12T12:32:08.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836491 2021-05-12T12:32:08.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491' 2021-05-12T12:32:08.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:32:08.229 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:32:08.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672969 2021-05-12T12:32:08.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672969 2021-05-12T12:32:08.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2021-05-12T12:32:08.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:32:08.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-05-12T12:32:08.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:32:08.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:32:08.360 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836491 2021-05-12T12:32:08.360 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:32:08.360 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836491 2021-05-12T12:32:08.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836491 2021-05-12T12:32:08.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-05-12T12:32:08.362 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:32:08.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836491 2021-05-12T12:32:08.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:32:09.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:32:09.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:32:10.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836491 2021-05-12T12:32:10.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:32:10.086 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-05-12T12:32:10.086 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:32:10.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:32:10.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672969 2021-05-12T12:32:10.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:32:10.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672969 2021-05-12T12:32:10.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-05-12T12:32:10.089 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949672969 2021-05-12T12:32:10.089 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:32:10.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672969 2021-05-12T12:32:10.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:32:10.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:32:10.451 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:32:10.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T12:32:10.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:32:10.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:32:10.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:32:10.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:32:10.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:32:10.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:32:10.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:32:11.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T12:32:11.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:32:11.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:32:11.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:32:11.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T12:32:11.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:32:11.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:32:11.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2021-05-12T12:32:11.786 INFO:tasks.workunit.client.0.smithi191.stderr:10+0 records in 2021-05-12T12:32:11.786 INFO:tasks.workunit.client.0.smithi191.stderr:10+0 records out 2021-05-12T12:32:11.795 INFO:tasks.workunit.client.0.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0453466 s, 231 MB/s 2021-05-12T12:32:11.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: p=1 2021-05-12T12:32:11.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-05-12T12:32:11.796 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-05-12T12:32:11.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:11.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2021-05-12T12:32:12.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:12.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2021-05-12T12:32:12.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:12.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2021-05-12T12:32:12.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:12.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2021-05-12T12:32:13.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:13.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2021-05-12T12:32:13.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:13.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2021-05-12T12:32:14.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:14.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2021-05-12T12:32:14.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:14.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2021-05-12T12:32:14.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:14.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2021-05-12T12:32:15.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:15.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2021-05-12T12:32:15.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:15.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2021-05-12T12:32:15.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:15.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2021-05-12T12:32:16.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:16.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2021-05-12T12:32:17.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:17.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2021-05-12T12:32:17.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:17.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2021-05-12T12:32:17.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:17.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2021-05-12T12:32:18.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:18.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2021-05-12T12:32:18.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:18.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2021-05-12T12:32:18.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:18.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2021-05-12T12:32:19.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:19.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2021-05-12T12:32:20.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:20.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2021-05-12T12:32:20.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:20.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2021-05-12T12:32:20.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:20.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2021-05-12T12:32:21.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:21.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2021-05-12T12:32:22.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:22.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2021-05-12T12:32:22.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:22.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2021-05-12T12:32:22.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:22.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2021-05-12T12:32:23.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:23.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2021-05-12T12:32:24.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:24.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2021-05-12T12:32:24.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:24.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2021-05-12T12:32:24.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:24.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2021-05-12T12:32:25.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:25.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2021-05-12T12:32:25.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:25.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2021-05-12T12:32:25.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:25.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2021-05-12T12:32:26.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:26.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2021-05-12T12:32:26.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:26.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2021-05-12T12:32:27.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:27.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2021-05-12T12:32:28.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:28.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2021-05-12T12:32:28.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:28.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2021-05-12T12:32:28.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:28.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2021-05-12T12:32:28.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:28.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2021-05-12T12:32:29.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:29.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2021-05-12T12:32:30.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:30.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2021-05-12T12:32:30.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:30.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2021-05-12T12:32:31.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:31.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2021-05-12T12:32:31.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:31.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2021-05-12T12:32:31.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:31.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2021-05-12T12:32:32.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:32.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2021-05-12T12:32:33.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:33.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2021-05-12T12:32:34.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:34.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2021-05-12T12:32:34.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:34.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2021-05-12T12:32:34.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:34.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2021-05-12T12:32:34.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:35.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2021-05-12T12:32:35.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:35.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2021-05-12T12:32:35.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:35.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2021-05-12T12:32:36.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:36.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2021-05-12T12:32:36.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:36.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2021-05-12T12:32:37.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:37.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2021-05-12T12:32:37.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:37.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2021-05-12T12:32:37.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:37.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2021-05-12T12:32:38.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:38.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2021-05-12T12:32:38.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:38.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2021-05-12T12:32:38.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:38.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2021-05-12T12:32:39.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:39.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2021-05-12T12:32:40.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:40.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2021-05-12T12:32:40.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:40.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2021-05-12T12:32:40.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:40.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2021-05-12T12:32:41.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:41.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2021-05-12T12:32:41.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:41.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2021-05-12T12:32:41.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:41.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2021-05-12T12:32:42.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:42.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2021-05-12T12:32:44.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:44.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2021-05-12T12:32:44.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:44.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2021-05-12T12:32:45.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:45.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2021-05-12T12:32:45.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:45.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2021-05-12T12:32:46.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:46.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2021-05-12T12:32:46.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:46.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2021-05-12T12:32:46.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:46.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2021-05-12T12:32:47.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:47.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2021-05-12T12:32:47.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:47.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2021-05-12T12:32:47.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:47.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2021-05-12T12:32:48.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:48.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2021-05-12T12:32:49.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:49.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2021-05-12T12:32:49.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:49.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2021-05-12T12:32:49.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:49.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2021-05-12T12:32:50.141 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:50.141 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2021-05-12T12:32:50.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:50.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2021-05-12T12:32:50.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:50.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2021-05-12T12:32:51.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:51.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2021-05-12T12:32:51.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:51.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2021-05-12T12:32:52.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:52.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2021-05-12T12:32:52.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:52.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2021-05-12T12:32:53.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:53.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2021-05-12T12:32:53.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:53.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2021-05-12T12:32:54.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:54.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2021-05-12T12:32:55.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:55.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2021-05-12T12:32:56.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:56.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2021-05-12T12:32:56.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:56.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2021-05-12T12:32:56.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:56.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2021-05-12T12:32:57.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:57.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2021-05-12T12:32:57.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:57.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2021-05-12T12:32:58.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:58.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2021-05-12T12:32:58.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:58.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2021-05-12T12:32:59.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:59.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2021-05-12T12:33:00.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:00.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2021-05-12T12:33:00.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:00.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2021-05-12T12:33:01.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:01.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2021-05-12T12:33:01.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:01.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2021-05-12T12:33:02.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:02.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2021-05-12T12:33:02.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:02.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2021-05-12T12:33:02.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:02.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2021-05-12T12:33:02.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:02.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2021-05-12T12:33:03.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:03.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2021-05-12T12:33:04.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:04.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2021-05-12T12:33:04.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:04.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2021-05-12T12:33:05.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:05.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2021-05-12T12:33:05.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:05.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2021-05-12T12:33:05.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:05.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2021-05-12T12:33:06.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:06.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2021-05-12T12:33:07.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:07.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2021-05-12T12:33:07.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:07.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2021-05-12T12:33:08.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:08.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2021-05-12T12:33:08.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:08.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2021-05-12T12:33:08.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:08.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2021-05-12T12:33:09.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:09.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2021-05-12T12:33:09.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:09.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2021-05-12T12:33:10.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:10.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2021-05-12T12:33:10.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:10.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2021-05-12T12:33:11.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:11.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2021-05-12T12:33:11.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:11.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2021-05-12T12:33:11.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:11.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2021-05-12T12:33:12.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:12.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2021-05-12T12:33:12.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:12.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2021-05-12T12:33:12.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:12.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2021-05-12T12:33:13.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:13.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2021-05-12T12:33:14.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:14.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2021-05-12T12:33:14.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:14.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2021-05-12T12:33:14.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:14.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2021-05-12T12:33:15.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:15.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2021-05-12T12:33:15.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:15.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2021-05-12T12:33:16.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:16.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2021-05-12T12:33:17.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:17.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2021-05-12T12:33:18.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:18.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2021-05-12T12:33:18.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:18.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2021-05-12T12:33:18.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:18.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2021-05-12T12:33:19.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:19.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2021-05-12T12:33:19.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:19.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2021-05-12T12:33:20.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:20.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2021-05-12T12:33:20.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:20.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2021-05-12T12:33:20.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:20.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2021-05-12T12:33:21.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:21.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2021-05-12T12:33:21.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:21.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2021-05-12T12:33:21.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:21.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2021-05-12T12:33:22.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:22.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2021-05-12T12:33:23.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:23.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2021-05-12T12:33:23.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:23.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2021-05-12T12:33:23.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:23.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2021-05-12T12:33:24.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:24.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2021-05-12T12:33:24.654 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:24.654 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2021-05-12T12:33:24.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:24.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2021-05-12T12:33:25.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:25.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2021-05-12T12:33:26.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:26.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2021-05-12T12:33:26.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:26.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2021-05-12T12:33:26.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:26.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2021-05-12T12:33:27.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:27.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2021-05-12T12:33:28.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:28.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2021-05-12T12:33:29.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:29.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2021-05-12T12:33:29.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:29.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2021-05-12T12:33:29.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:29.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2021-05-12T12:33:30.141 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:30.142 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2021-05-12T12:33:30.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:30.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2021-05-12T12:33:30.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:30.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2021-05-12T12:33:31.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:31.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2021-05-12T12:33:32.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:32.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2021-05-12T12:33:32.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:32.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2021-05-12T12:33:32.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:32.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2021-05-12T12:33:33.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:33.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2021-05-12T12:33:33.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:33.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2021-05-12T12:33:34.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:34.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2021-05-12T12:33:34.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:34.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2021-05-12T12:33:35.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:35.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2021-05-12T12:33:35.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:35.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2021-05-12T12:33:35.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:35.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2021-05-12T12:33:36.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:36.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2021-05-12T12:33:36.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:36.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2021-05-12T12:33:36.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:36.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2021-05-12T12:33:37.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:37.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2021-05-12T12:33:37.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:37.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2021-05-12T12:33:38.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:38.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2021-05-12T12:33:39.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:39.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2021-05-12T12:33:41.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:41.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2021-05-12T12:33:41.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:41.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2021-05-12T12:33:42.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:42.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2021-05-12T12:33:42.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:42.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2021-05-12T12:33:42.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:42.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2021-05-12T12:33:43.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:43.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2021-05-12T12:33:43.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:43.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2021-05-12T12:33:44.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:44.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2021-05-12T12:33:44.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:44.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2021-05-12T12:33:44.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:44.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2021-05-12T12:33:45.316 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 1 + 1 2021-05-12T12:33:45.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=2 2021-05-12T12:33:45.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-05-12T12:33:45.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-05-12T12:33:45.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:45.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2021-05-12T12:33:45.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:45.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2021-05-12T12:33:46.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:46.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2021-05-12T12:33:46.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:46.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2021-05-12T12:33:46.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:46.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2021-05-12T12:33:47.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:47.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2021-05-12T12:33:47.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:47.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2021-05-12T12:33:48.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:48.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2021-05-12T12:33:48.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:48.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2021-05-12T12:33:48.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:48.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2021-05-12T12:33:49.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:49.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2021-05-12T12:33:50.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:50.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2021-05-12T12:33:50.434 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:50.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2021-05-12T12:33:50.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:50.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2021-05-12T12:33:51.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:51.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2021-05-12T12:33:51.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:51.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2021-05-12T12:33:51.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:51.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2021-05-12T12:33:52.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:52.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2021-05-12T12:33:53.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:53.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2021-05-12T12:33:53.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:53.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2021-05-12T12:33:53.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:53.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2021-05-12T12:33:54.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:54.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2021-05-12T12:33:54.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:54.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2021-05-12T12:33:54.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:54.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2021-05-12T12:33:55.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:55.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2021-05-12T12:33:56.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:56.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2021-05-12T12:33:56.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:56.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2021-05-12T12:33:57.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:57.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2021-05-12T12:33:57.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:57.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2021-05-12T12:33:58.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:58.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2021-05-12T12:33:58.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:58.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2021-05-12T12:33:58.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:58.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2021-05-12T12:33:58.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:58.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2021-05-12T12:33:59.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:59.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2021-05-12T12:34:00.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:00.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2021-05-12T12:34:00.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:00.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2021-05-12T12:34:01.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:01.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2021-05-12T12:34:01.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:01.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2021-05-12T12:34:01.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:01.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2021-05-12T12:34:02.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:02.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2021-05-12T12:34:03.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:03.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2021-05-12T12:34:03.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:03.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2021-05-12T12:34:03.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:03.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2021-05-12T12:34:04.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:04.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2021-05-12T12:34:04.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:04.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2021-05-12T12:34:04.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:04.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2021-05-12T12:34:05.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:05.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2021-05-12T12:34:06.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:06.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2021-05-12T12:34:07.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:07.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2021-05-12T12:34:07.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:07.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2021-05-12T12:34:07.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:07.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2021-05-12T12:34:08.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:08.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2021-05-12T12:34:08.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:08.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2021-05-12T12:34:08.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:08.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2021-05-12T12:34:09.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:09.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2021-05-12T12:34:10.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:10.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2021-05-12T12:34:10.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:10.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2021-05-12T12:34:10.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:10.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2021-05-12T12:34:10.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:10.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2021-05-12T12:34:11.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:11.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2021-05-12T12:34:11.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:11.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2021-05-12T12:34:12.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:12.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2021-05-12T12:34:13.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:13.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2021-05-12T12:34:13.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:13.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2021-05-12T12:34:13.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:13.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2021-05-12T12:34:14.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:14.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2021-05-12T12:34:14.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:14.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2021-05-12T12:34:14.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:14.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2021-05-12T12:34:14.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:14.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2021-05-12T12:34:16.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:16.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2021-05-12T12:34:16.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:16.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2021-05-12T12:34:17.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:17.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2021-05-12T12:34:17.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:17.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2021-05-12T12:34:17.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:17.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2021-05-12T12:34:18.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:18.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2021-05-12T12:34:18.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:18.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2021-05-12T12:34:19.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:19.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2021-05-12T12:34:19.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:19.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2021-05-12T12:34:19.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:19.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2021-05-12T12:34:20.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:20.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2021-05-12T12:34:20.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:20.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2021-05-12T12:34:20.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:20.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2021-05-12T12:34:21.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:21.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2021-05-12T12:34:22.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:22.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2021-05-12T12:34:22.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:22.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2021-05-12T12:34:22.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:22.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2021-05-12T12:34:23.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:23.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2021-05-12T12:34:23.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:23.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2021-05-12T12:34:23.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:23.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2021-05-12T12:34:24.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:24.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2021-05-12T12:34:25.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:25.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2021-05-12T12:34:25.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:25.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2021-05-12T12:34:25.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:25.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2021-05-12T12:34:26.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:26.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2021-05-12T12:34:26.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:26.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2021-05-12T12:34:28.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:28.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2021-05-12T12:34:28.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:28.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2021-05-12T12:34:28.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:28.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2021-05-12T12:34:29.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:29.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2021-05-12T12:34:29.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:29.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2021-05-12T12:34:29.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:29.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2021-05-12T12:34:30.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:30.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2021-05-12T12:34:30.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:30.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2021-05-12T12:34:32.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:32.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2021-05-12T12:34:32.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:32.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2021-05-12T12:34:33.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:33.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2021-05-12T12:34:33.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:33.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2021-05-12T12:34:34.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:34.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2021-05-12T12:34:34.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:34.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2021-05-12T12:34:34.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:34.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2021-05-12T12:34:35.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:35.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2021-05-12T12:34:35.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:35.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2021-05-12T12:34:35.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:35.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2021-05-12T12:34:36.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:36.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2021-05-12T12:34:36.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:36.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2021-05-12T12:34:37.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:37.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2021-05-12T12:34:37.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:37.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2021-05-12T12:34:38.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:38.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2021-05-12T12:34:38.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:38.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2021-05-12T12:34:39.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:39.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2021-05-12T12:34:39.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:39.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2021-05-12T12:34:40.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:40.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2021-05-12T12:34:40.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:40.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2021-05-12T12:34:41.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:41.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2021-05-12T12:34:41.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:41.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2021-05-12T12:34:42.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:42.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2021-05-12T12:34:42.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:42.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2021-05-12T12:34:42.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:42.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2021-05-12T12:34:43.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:43.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2021-05-12T12:34:43.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:43.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2021-05-12T12:34:43.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:43.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2021-05-12T12:34:44.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:44.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2021-05-12T12:34:45.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:45.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2021-05-12T12:34:45.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:45.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2021-05-12T12:34:45.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:45.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2021-05-12T12:34:46.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:46.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2021-05-12T12:34:46.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:46.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2021-05-12T12:34:46.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:46.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2021-05-12T12:34:47.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:47.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2021-05-12T12:34:48.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:48.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2021-05-12T12:34:48.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:48.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2021-05-12T12:34:49.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:49.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2021-05-12T12:34:49.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:49.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2021-05-12T12:34:49.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:49.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2021-05-12T12:34:50.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:50.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2021-05-12T12:34:51.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:51.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2021-05-12T12:34:51.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:51.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2021-05-12T12:34:51.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:51.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2021-05-12T12:34:52.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:52.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2021-05-12T12:34:52.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:52.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2021-05-12T12:34:52.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:52.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2021-05-12T12:34:53.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:53.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2021-05-12T12:34:54.141 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:54.142 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2021-05-12T12:34:54.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:54.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2021-05-12T12:34:54.759 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:54.759 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2021-05-12T12:34:55.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:55.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2021-05-12T12:34:55.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:55.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2021-05-12T12:34:55.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:55.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2021-05-12T12:34:56.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:56.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2021-05-12T12:34:57.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:57.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2021-05-12T12:34:57.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:57.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2021-05-12T12:34:57.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:57.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2021-05-12T12:34:58.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:58.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2021-05-12T12:34:58.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:58.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2021-05-12T12:34:58.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:58.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2021-05-12T12:35:00.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:00.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2021-05-12T12:35:00.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:00.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2021-05-12T12:35:01.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:01.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2021-05-12T12:35:01.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:01.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2021-05-12T12:35:01.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:01.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2021-05-12T12:35:02.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:02.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2021-05-12T12:35:02.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:02.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2021-05-12T12:35:03.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:03.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2021-05-12T12:35:03.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:03.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2021-05-12T12:35:04.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:04.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2021-05-12T12:35:04.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:04.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2021-05-12T12:35:04.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:04.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2021-05-12T12:35:05.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:05.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2021-05-12T12:35:05.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:05.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2021-05-12T12:35:05.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:05.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2021-05-12T12:35:06.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:06.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2021-05-12T12:35:07.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:07.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2021-05-12T12:35:07.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:07.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2021-05-12T12:35:07.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:07.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2021-05-12T12:35:08.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:08.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2021-05-12T12:35:08.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:08.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2021-05-12T12:35:08.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:08.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2021-05-12T12:35:09.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:09.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2021-05-12T12:35:10.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:10.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2021-05-12T12:35:11.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:11.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2021-05-12T12:35:13.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:13.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2021-05-12T12:35:13.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:13.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2021-05-12T12:35:13.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:13.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2021-05-12T12:35:14.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:14.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2021-05-12T12:35:14.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:14.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2021-05-12T12:35:14.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:14.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2021-05-12T12:35:15.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:15.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2021-05-12T12:35:16.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:16.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2021-05-12T12:35:16.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:16.434 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2021-05-12T12:35:16.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:16.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2021-05-12T12:35:17.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 2 + 1 2021-05-12T12:35:17.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=3 2021-05-12T12:35:17.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2021-05-12T12:35:17.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local poolname=test1 2021-05-12T12:35:17.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local objectname=obj1-p1 2021-05-12T12:35:17.073 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: get_primary test1 obj1-p1 2021-05-12T12:35:17.073 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local poolname=test1 2021-05-12T12:35:17.073 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: local objectname=obj1-p1 2021-05-12T12:35:17.073 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T12:35:17.074 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: jq .acting_primary 2021-05-12T12:35:17.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local primary=1 2021-05-12T12:35:17.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T12:35:17.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1173: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-05-12T12:35:17.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: local otherosd=null 2021-05-12T12:35:17.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:423: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T12:35:18.215 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:35:18.215 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:32:05.730832+0000 38'600 38:644 [1] 1 [1] 1 0'0 2021-05-12T12:31:46.974016+0000 0'0 2021-05-12T12:31:46.974016+0000 0 2021-05-12T12:35:18.215 INFO:tasks.workunit.client.0.smithi191.stdout:2.0 190 0 0 0 0 1990197248 0 0 569 569 active+clean 2021-05-12T12:32:07.169517+0000 38'569 38:608 [0] 0 [0] 0 0'0 2021-05-12T12:31:49.263826+0000 0'0 2021-05-12T12:31:49.263826+0000 0 2021-05-12T12:35:18.216 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:35:18.216 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:35:18.216 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:35:18.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: ERRORS=0 2021-05-12T12:35:18.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-05-12T12:35:19.032 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 size to 2 2021-05-12T12:35:19.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-05-12T12:35:19.759 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 2 size to 2 2021-05-12T12:35:19.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T12:35:20.113 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:35:20.114 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:32:05.730832+0000 38'600 38:644 [1] 1 [1] 1 0'0 2021-05-12T12:31:46.974016+0000 0'0 2021-05-12T12:31:46.974016+0000 0 2021-05-12T12:35:20.114 INFO:tasks.workunit.client.0.smithi191.stdout:2.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:32:07.169517+0000 38'600 38:639 [0] 0 [0] 0 0'0 2021-05-12T12:31:49.263826+0000 0'0 2021-05-12T12:31:49.263826+0000 0 2021-05-12T12:35:20.114 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:35:20.114 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:35:20.115 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:35:20.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: set -o pipefail 2021-05-12T12:35:20.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:434: TEST_recovery_pool_priority: count=0 2021-05-12T12:35:20.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T12:35:20.126 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T12:35:20.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T12:35:20.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T12:35:20.470 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:35:20.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 0 == 2 2021-05-12T12:35:20.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-05-12T12:35:22.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 0 -eq 10 2021-05-12T12:35:22.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 0 + 1 2021-05-12T12:35:22.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=1 2021-05-12T12:35:22.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T12:35:22.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T12:35:22.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T12:35:22.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T12:35:22.823 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:35:22.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 0 == 2 2021-05-12T12:35:22.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-05-12T12:35:24.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 1 -eq 10 2021-05-12T12:35:24.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 1 + 1 2021-05-12T12:35:24.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=2 2021-05-12T12:35:24.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T12:35:24.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T12:35:24.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T12:35:24.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T12:35:25.177 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:35:25.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 2 == 2 2021-05-12T12:35:25.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:440: TEST_recovery_pool_priority: break 2021-05-12T12:35:25.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:450: TEST_recovery_pool_priority: set +o pipefail 2021-05-12T12:35:25.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T12:35:25.537 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:35:25.538 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T12:35:19.966625+0000 38'600 43:658 [1,0] 1 [1] 1 0'0 2021-05-12T12:31:46.974016+0000 0'0 2021-05-12T12:31:46.974016+0000 0 2021-05-12T12:35:25.538 INFO:tasks.workunit.client.0.smithi191.stdout:2.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T12:35:20.945931+0000 38'600 44:652 [0,1] 0 [0] 0 0'0 2021-05-12T12:31:49.263826+0000 0'0 2021-05-12T12:31:49.263826+0000 0 2021-05-12T12:35:25.539 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:35:25.539 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:35:25.539 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:35:25.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: get_asok_path osd.1 2021-05-12T12:35:25.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:35:25.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:35:25.552 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:35:25.552 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:35:25.553 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:35:25.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:35:25.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: CEPH_ARGS= 2021-05-12T12:35:25.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:35:25.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: echo osd.1 2021-05-12T12:35:25.653 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 2021-05-12T12:35:25.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:455: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:25.654 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:35:25.654 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:35:25.654 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:35:25.654 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:35:25.654 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:35:25.655 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:35:25.655 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:35:25.655 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:35:25.655 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 191, 2021-05-12T12:35:25.656 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:35:25.656 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:35:25.656 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:35:25.656 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:35:25.656 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:35:25.657 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:35:25.657 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:35:25.657 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:35:25.657 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:35:25.658 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:35:25.658 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "2.0", 2021-05-12T12:35:25.658 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 192, 2021-05-12T12:35:25.658 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:35:25.658 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:35:25.659 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:35:25.659 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:35:25.659 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:35:25.660 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.0 2021-05-12T12:35:25.661 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:35:25.661 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:35:25.661 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:35:25.661 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:35:25.662 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:35:25.662 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.0.asok 2021-05-12T12:35:25.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2021-05-12T12:35:25.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.0.asok dump_recovery_reservations 2021-05-12T12:35:25.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.0 2021-05-12T12:35:25.757 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 2021-05-12T12:35:25.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:25.758 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:35:25.758 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:35:25.759 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:35:25.759 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:35:25.760 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:35:25.760 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:35:25.760 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:35:25.760 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "2.0", 2021-05-12T12:35:25.761 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 192, 2021-05-12T12:35:25.761 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:35:25.761 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:35:25.761 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:35:25.761 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:35:25.762 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:35:25.762 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:35:25.762 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:35:25.762 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:35:25.763 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:35:25.763 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:35:25.763 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:35:25.763 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 191, 2021-05-12T12:35:25.764 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:35:25.764 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:35:25.764 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:35:25.764 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:35:25.764 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:35:25.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:25.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:35:25.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-05-12T12:35:25.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: ITEM=1.0 2021-05-12T12:35:25.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:462: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-05-12T12:35:25.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:25.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-05-12T12:35:25.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: PRIO=191 2021-05-12T12:35:25.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:468: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-05-12T12:35:25.768 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:25.768 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-05-12T12:35:25.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-05-12T12:35:25.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: ITEM=1.0 2021-05-12T12:35:25.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:477: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-05-12T12:35:25.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:25.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-05-12T12:35:25.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: PRIO=191 2021-05-12T12:35:25.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:483: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-05-12T12:35:25.772 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:25.772 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:35:25.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-05-12T12:35:25.774 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: ITEM=2.0 2021-05-12T12:35:25.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:492: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-05-12T12:35:25.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:25.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-05-12T12:35:25.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: PRIO=192 2021-05-12T12:35:25.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:498: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-05-12T12:35:25.778 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:25.778 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-05-12T12:35:25.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-05-12T12:35:25.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: ITEM=2.0 2021-05-12T12:35:25.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:507: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-05-12T12:35:25.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:25.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-05-12T12:35:25.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: PRIO=192 2021-05-12T12:35:25.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:513: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-05-12T12:35:25.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:520: TEST_recovery_pool_priority: wait_for_clean 2021-05-12T12:35:25.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:35:25.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:35:25.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:35:25.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:35:25.787 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:35:25.787 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:35:25.787 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:35:25.787 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:35:25.788 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:35:25.788 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:35:25.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:35:25.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:35:25.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:35:25.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:35:25.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:35:26.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:35:26.234 INFO:tasks.workunit.client.0.smithi191.stderr:1' 2021-05-12T12:35:26.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:35:26.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:35:26.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:35:26.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836532 2021-05-12T12:35:26.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836532 2021-05-12T12:35:26.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836532' 2021-05-12T12:35:26.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:35:26.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:35:26.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949673010 2021-05-12T12:35:26.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949673010 2021-05-12T12:35:26.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836532 1-42949673010' 2021-05-12T12:35:26.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:35:26.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836532 2021-05-12T12:35:26.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:35:26.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:35:26.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836532 2021-05-12T12:35:26.589 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:35:26.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836532 2021-05-12T12:35:26.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2021-05-12T12:35:26.591 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836532 2021-05-12T12:35:26.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:35:26.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836530 -lt 21474836532 2021-05-12T12:35:26.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:35:27.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:35:27.973 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:35:28.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836532 -lt 21474836532 2021-05-12T12:35:28.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:35:28.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673010 2021-05-12T12:35:28.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:35:28.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:35:28.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949673010 2021-05-12T12:35:28.339 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:35:28.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949673010 2021-05-12T12:35:28.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2021-05-12T12:35:28.340 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 42949673010 2021-05-12T12:35:28.340 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:35:28.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949673010 -lt 42949673010 2021-05-12T12:35:28.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:35:28.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:28.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:29.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T12:35:29.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:29.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:29.228 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:29.228 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:29.229 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:29.229 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:29.229 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:29.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:29.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:29.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:29.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:30.056 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:30.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:35:30.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:35:30.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:35:30.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:30.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:30.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:30.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:30.158 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:30.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:30.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:30.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:30.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:30.160 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:30.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:30.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:30.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:30.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:30.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:30.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:30.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:30.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:30.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:30.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:30.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:30.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:30.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:31.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2919362 2021-05-12T12:35:31.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2919362 '!=' null 2021-05-12T12:35:31.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:31.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:31.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:31.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:31.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:31.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:31.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:31.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:31.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:31.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:31.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:31.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:31.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:31.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:31.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:32.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:32.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:32.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:32.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:32.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:32.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:32.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:32.434 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:32.434 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:32.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2919362 2021-05-12T12:35:32.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2919362 '!=' null 2021-05-12T12:35:32.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:32.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:32.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:33.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:33.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:33.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:33.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:33.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:33.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:33.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:33.022 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:33.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:33.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:33.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:33.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:33.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:33.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:33.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:33.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:33.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:33.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:33.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:33.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:33.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:34.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368252 2021-05-12T12:35:34.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368252 '!=' null 2021-05-12T12:35:34.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:34.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:34.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:34.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:34.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:34.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:34.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:34.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:34.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:34.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:34.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:34.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:34.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:34.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:34.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:35.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:35.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:35.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:35.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:35.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:35.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:35.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:35.451 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:35.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:35.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368277 2021-05-12T12:35:35.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368277 '!=' null 2021-05-12T12:35:35.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:35.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:35.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:36.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:36.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:36.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:36.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:36.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:36.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:36.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:36.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:36.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:36.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:36.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:36.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:36.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:36.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:36.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:36.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:36.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:36.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:36.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:36.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:36.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:37.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368305 2021-05-12T12:35:37.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368305 '!=' null 2021-05-12T12:35:37.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:37.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:37.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:37.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:37.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:37.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:37.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:37.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:37.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:37.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:37.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:37.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:37.842 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:37.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:37.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:38.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:38.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:38.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:38.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:38.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:38.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:38.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:38.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:38.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:38.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368305 2021-05-12T12:35:38.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368305 '!=' null 2021-05-12T12:35:38.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:38.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:38.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:38.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:38.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:38.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:38.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:38.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:38.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:38.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:38.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:39.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:39.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:39.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:39.293 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:39.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:39.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:39.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:39.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:39.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:39.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:39.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:39.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:39.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:40.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241975 2021-05-12T12:35:40.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241975 '!=' null 2021-05-12T12:35:40.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:40.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:40.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:40.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:40.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:40.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:40.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:40.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:40.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:40.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:40.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:40.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:40.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:40.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:40.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:41.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:41.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:41.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:41.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:41.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:41.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:41.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:41.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:41.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:41.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494678 2021-05-12T12:35:41.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494678 '!=' null 2021-05-12T12:35:41.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:41.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:41.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:41.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:41.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:41.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:41.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:41.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:41.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:41.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:41.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:42.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:42.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:42.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:42.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:42.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:42.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:42.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:42.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:42.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:42.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:42.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:42.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:42.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:43.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494678 2021-05-12T12:35:43.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494678 '!=' null 2021-05-12T12:35:43.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:43.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:43.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:43.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:43.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:43.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:43.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:43.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:43.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:43.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:43.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:43.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:43.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:43.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:43.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:44.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:44.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:44.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:44.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:44.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:44.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:44.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:44.126 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:44.126 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:44.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115730 2021-05-12T12:35:44.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115730 '!=' null 2021-05-12T12:35:44.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:44.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:44.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:44.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:44.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:44.785 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:44.785 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:44.785 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:44.785 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:44.786 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:44.786 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:45.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:45.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:45.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:45.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:45.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:45.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:45.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:45.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:45.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:45.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:45.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:45.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:45.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:46.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368375 2021-05-12T12:35:46.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368375 '!=' null 2021-05-12T12:35:46.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:46.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:46.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:46.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:46.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:46.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:46.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:46.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:46.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:46.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:46.211 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:46.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:46.569 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:46.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:46.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:47.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:47.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:47.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:47.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:47.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:47.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:47.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:47.053 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:47.053 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:47.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368371 2021-05-12T12:35:47.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368371 '!=' null 2021-05-12T12:35:47.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:47.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:47.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:47.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:47.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:47.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:47.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:47.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:47.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:47.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:47.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:47.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:47.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:47.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:48.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:48.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:48.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:48.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:48.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:48.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:48.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:48.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:48.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:48.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:48.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368371 2021-05-12T12:35:48.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368371 '!=' null 2021-05-12T12:35:48.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:48.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:48.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:49.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:49.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:49.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:49.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:49.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:49.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:49.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:49.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:49.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:49.433 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:49.433 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:49.433 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:49.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:49.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:49.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:49.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:49.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:49.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:49.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:49.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:49.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:50.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115726 2021-05-12T12:35:50.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115726 '!=' null 2021-05-12T12:35:50.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:50.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:50.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:50.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:50.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:50.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:50.522 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:50.522 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:50.522 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:50.522 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:50.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:50.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:50.882 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:50.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:50.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:51.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:51.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:51.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:51.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:51.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:51.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:51.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:51.397 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:51.397 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:51.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368357 2021-05-12T12:35:51.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368357 '!=' null 2021-05-12T12:35:51.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:51.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:51.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:51.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:51.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:51.991 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:51.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:51.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:51.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:51.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:51.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:52.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:52.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:52.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:52.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:52.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:52.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:52.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:52.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:52.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:52.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:52.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:52.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:52.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:53.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115722 2021-05-12T12:35:53.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115722 '!=' null 2021-05-12T12:35:53.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:53.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:53.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:53.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:53.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:53.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:53.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:53.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:53.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:53.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:53.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:53.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:53.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:53.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:53.812 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:54.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:54.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:54.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:54.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:54.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:54.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:54.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:54.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:54.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:54.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115722 2021-05-12T12:35:54.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115722 '!=' null 2021-05-12T12:35:54.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:54.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:54.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:54.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:54.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:54.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:54.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:54.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:54.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:54.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:54.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:55.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:55.246 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:55.246 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:55.247 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:55.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:55.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:55.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:55.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:55.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:55.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:55.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:55.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:55.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:56.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494686 2021-05-12T12:35:56.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494686 '!=' null 2021-05-12T12:35:56.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:56.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:56.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:56.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:56.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:56.354 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:56.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:56.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:56.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:56.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:56.356 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:56.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:56.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:56.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:56.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:57.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:57.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:57.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:57.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:57.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:57.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:57.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:57.208 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:57.208 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:57.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494695 2021-05-12T12:35:57.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494695 '!=' null 2021-05-12T12:35:57.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:57.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:57.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:57.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:57.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:57.830 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:57.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:57.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:57.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:57.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:57.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:58.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:58.190 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:58.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:58.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:58.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:58.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:58.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:58.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:58.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:58.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:58.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:58.675 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:58.675 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:59.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494695 2021-05-12T12:35:59.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494695 '!=' null 2021-05-12T12:35:59.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:59.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:59.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:59.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:59.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:59.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:59.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:59.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:59.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:59.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:59.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:59.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:59.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:59.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:59.622 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:00.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:00.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:00.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:00.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:00.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:00.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:00.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:00.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:00.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:00.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242003 2021-05-12T12:36:00.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242003 '!=' null 2021-05-12T12:36:00.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:00.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:00.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:00.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:00.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:00.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:00.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:00.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:00.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:00.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:00.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:01.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:01.043 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:01.043 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:01.043 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:01.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:01.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:01.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:01.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:01.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:01.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:01.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:01.560 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:01.561 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:02.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494631 2021-05-12T12:36:02.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494631 '!=' null 2021-05-12T12:36:02.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:02.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:02.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:02.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:02.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:02.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:02.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:02.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:02.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:02.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:02.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:02.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:02.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:02.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:02.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:03.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:03.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:03.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:03.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:03.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:03.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:03.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:03.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:03.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:03.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115667 2021-05-12T12:36:03.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115667 '!=' null 2021-05-12T12:36:03.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:03.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:03.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:03.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:03.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:03.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:03.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:03.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:03.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:03.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:03.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:03.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:03.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:03.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:03.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:04.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:04.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:04.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:04.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:04.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:04.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:04.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:04.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:04.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:04.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115667 2021-05-12T12:36:04.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115667 '!=' null 2021-05-12T12:36:04.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:04.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:04.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:05.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:05.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:05.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:05.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:05.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:05.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:05.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:05.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:05.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:05.451 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:05.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:05.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:05.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:05.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:05.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:05.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:05.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:05.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:05.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:05.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:05.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:06.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368312 2021-05-12T12:36:06.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368312 '!=' null 2021-05-12T12:36:06.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:06.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:06.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:06.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:06.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:06.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:06.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:06.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:06.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:06.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:06.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:06.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:06.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:06.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:06.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:07.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:07.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:07.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:07.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:07.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:07.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:07.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:07.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:07.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:07.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T12:36:07.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T12:36:07.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:07.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:07.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:08.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:08.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:08.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:08.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:08.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:08.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:08.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:08.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:08.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:08.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:08.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:08.356 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:08.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:08.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:08.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:08.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:08.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:08.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:08.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:08.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:08.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:09.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115572 2021-05-12T12:36:09.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115572 '!=' null 2021-05-12T12:36:09.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:09.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:09.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:09.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:09.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:09.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:09.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:09.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:09.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:09.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:09.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:09.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:09.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:09.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:09.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:10.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:10.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:10.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:10.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:10.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:10.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:10.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:10.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:10.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:10.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115572 2021-05-12T12:36:10.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115572 '!=' null 2021-05-12T12:36:10.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:10.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:10.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:10.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:10.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:10.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:10.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:10.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:10.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:10.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:10.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:11.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:11.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:11.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:11.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:11.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:11.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:11.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:11.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:11.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:11.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:11.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:11.722 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:11.722 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:12.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368269 2021-05-12T12:36:12.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368269 '!=' null 2021-05-12T12:36:12.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:12.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:12.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:12.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:12.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:12.405 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:12.406 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:12.406 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:12.406 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:12.406 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:12.407 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:12.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:12.771 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:12.772 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:12.772 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:13.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:13.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:13.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:13.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:13.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:13.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:13.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:13.262 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:13.262 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:13.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115656 2021-05-12T12:36:13.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115656 '!=' null 2021-05-12T12:36:13.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:13.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:13.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:13.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:13.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:13.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:13.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:13.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:13.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:13.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:13.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:14.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:14.214 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:14.214 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:14.214 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:14.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:14.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:14.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:14.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:14.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:14.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:14.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:14.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:14.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:15.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115656 2021-05-12T12:36:15.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115656 '!=' null 2021-05-12T12:36:15.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:15.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:15.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:15.303 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:15.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:15.305 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:15.305 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:15.305 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:15.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:15.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:15.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:15.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:15.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:15.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:15.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:16.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:16.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:16.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:16.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:16.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:16.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:16.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:16.160 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:16.160 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:16.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494652 2021-05-12T12:36:16.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494652 '!=' null 2021-05-12T12:36:16.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:16.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:16.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:16.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:16.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:16.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:16.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:16.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:16.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:16.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:16.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:17.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:17.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:17.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:17.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:17.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:17.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:17.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:17.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:17.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:17.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:17.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:17.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:17.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:18.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494665 2021-05-12T12:36:18.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494665 '!=' null 2021-05-12T12:36:18.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:18.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:18.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:18.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:18.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:18.204 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:18.204 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:18.204 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:18.205 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:18.205 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:18.205 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:18.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:18.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:18.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:18.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:19.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:19.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:19.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:19.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:19.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:19.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:19.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:19.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:19.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:19.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115658 2021-05-12T12:36:19.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115658 '!=' null 2021-05-12T12:36:19.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:19.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:19.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:19.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:19.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:19.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:19.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:19.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:19.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:19.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:19.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:19.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:19.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:19.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:19.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:20.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:20.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:20.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:20.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:20.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:20.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:20.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:20.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:20.487 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:20.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115658 2021-05-12T12:36:20.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115658 '!=' null 2021-05-12T12:36:20.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:20.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:20.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:21.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:21.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:21.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:21.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:21.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:21.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:21.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:21.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:21.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:21.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:21.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:21.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:21.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:21.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:21.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:21.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:21.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:21.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:21.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:21.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:21.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:22.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368360 2021-05-12T12:36:22.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368360 '!=' null 2021-05-12T12:36:22.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:22.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:22.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:22.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:22.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:22.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:22.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:22.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:22.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:22.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:22.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:22.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:22.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:22.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:22.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:23.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:23.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:23.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:23.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:23.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:23.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:23.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:23.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:23.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:23.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242034 2021-05-12T12:36:23.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242034 '!=' null 2021-05-12T12:36:23.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:23.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:23.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:23.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:23.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:23.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:23.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:23.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:23.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:23.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:23.983 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:24.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:24.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:24.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:24.342 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:24.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:24.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:24.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:24.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:24.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:24.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:24.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:24.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:24.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:25.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494662 2021-05-12T12:36:25.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494662 '!=' null 2021-05-12T12:36:25.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:25.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:25.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:25.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:25.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:25.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:25.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:25.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:25.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:25.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:25.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:25.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:25.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:25.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:25.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:26.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:26.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:26.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:26.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:26.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:26.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:26.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:26.313 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:26.313 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:26.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494662 2021-05-12T12:36:26.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494662 '!=' null 2021-05-12T12:36:26.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:26.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:26.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:26.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:26.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:26.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:26.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:26.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:26.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:26.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:26.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:27.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:27.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:27.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:27.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:27.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:27.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:27.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:27.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:27.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:27.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:27.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:27.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:27.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:28.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494659 2021-05-12T12:36:28.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494659 '!=' null 2021-05-12T12:36:28.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:28.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:28.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:28.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:28.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:28.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:28.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:28.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:28.396 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:28.396 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:28.396 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:28.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:28.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:28.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:28.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:29.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:29.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:29.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:29.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:29.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:29.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:29.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:29.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:29.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:29.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115646 2021-05-12T12:36:29.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115646 '!=' null 2021-05-12T12:36:29.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:29.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:29.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:29.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:29.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:29.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:29.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:29.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:29.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:29.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:29.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:30.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:30.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:30.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:30.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:30.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:30.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:30.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:30.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:30.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:30.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:30.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:30.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:30.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:31.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115646 2021-05-12T12:36:31.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115646 '!=' null 2021-05-12T12:36:31.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:31.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:31.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:31.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:31.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:31.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:31.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:31.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:31.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:31.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:31.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:31.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:31.631 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:31.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:31.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:32.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:32.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:32.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:32.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:32.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:32.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:32.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:32.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:32.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:32.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494625 2021-05-12T12:36:32.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494625 '!=' null 2021-05-12T12:36:32.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:32.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:32.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:32.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:32.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:32.717 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:32.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:32.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:32.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:32.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:32.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:33.088 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:33.089 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:33.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:33.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:33.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:33.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:33.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:33.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:33.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:33.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:33.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:33.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:33.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:34.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115597 2021-05-12T12:36:34.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115597 '!=' null 2021-05-12T12:36:34.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:34.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:34.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:34.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:34.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:34.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:34.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:34.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:34.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:34.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:34.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:34.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:34.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:34.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:34.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:35.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:35.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:35.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:35.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:35.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:35.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:35.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:35.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:35.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:35.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241938 2021-05-12T12:36:35.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241938 '!=' null 2021-05-12T12:36:35.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:35.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:35.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:35.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:35.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:35.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:35.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:35.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:35.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:35.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:35.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:36.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:36.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:36.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:36.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:36.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:36.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:36.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:36.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:36.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:36.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:36.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:36.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:36.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:37.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241938 2021-05-12T12:36:37.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241938 '!=' null 2021-05-12T12:36:37.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:37.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:37.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:37.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:37.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:37.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:37.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:37.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:37.119 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:37.119 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:37.119 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:37.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:37.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:37.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:37.633 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:38.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:38.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:38.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:38.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:38.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:38.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:38.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:38.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:38.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:38.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241950 2021-05-12T12:36:38.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241950 '!=' null 2021-05-12T12:36:38.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:38.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:38.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:38.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:38.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:38.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:38.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:38.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:38.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:38.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:38.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:38.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:38.973 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:38.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:38.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:39.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:39.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:39.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:39.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:39.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:39.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:39.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:39.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:39.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:39.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115622 2021-05-12T12:36:39.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115622 '!=' null 2021-05-12T12:36:39.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:39.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:39.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:40.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:40.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:40.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:40.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:40.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:40.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:40.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:40.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:40.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:40.410 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:40.410 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:40.411 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:40.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:40.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:40.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:40.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:40.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:40.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:40.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:40.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:40.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:41.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494637 2021-05-12T12:36:41.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494637 '!=' null 2021-05-12T12:36:41.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:41.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:41.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:41.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:41.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:41.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:41.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:41.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:41.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:41.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:41.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:41.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:41.862 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:41.862 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:41.863 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:42.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:42.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:42.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:42.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:42.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:42.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:42.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:42.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:42.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:42.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494637 2021-05-12T12:36:42.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494637 '!=' null 2021-05-12T12:36:42.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:42.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:42.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:42.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:42.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:42.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:42.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:42.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:42.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:42.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:42.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:43.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:43.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:43.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:43.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:43.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:43.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:43.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:43.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:43.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:43.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:43.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:43.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:43.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:44.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115673 2021-05-12T12:36:44.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115673 '!=' null 2021-05-12T12:36:44.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:44.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:44.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:44.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:44.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:44.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:44.404 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:44.404 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:44.404 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:44.404 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:44.405 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:44.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:44.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:44.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:44.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:45.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:45.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:45.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:45.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:45.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:45.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:45.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:45.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:45.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:45.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494672 2021-05-12T12:36:45.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494672 '!=' null 2021-05-12T12:36:45.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:45.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:45.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:45.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:45.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:45.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:45.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:45.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:45.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:45.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:45.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:46.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:46.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:46.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:46.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:46.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:46.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:46.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:46.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:46.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:46.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:46.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:46.743 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:46.743 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:47.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494687 2021-05-12T12:36:47.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494687 '!=' null 2021-05-12T12:36:47.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:47.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:47.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:47.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:47.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:47.318 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:47.318 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:47.318 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:47.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:47.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:47.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:47.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:47.712 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:47.712 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:47.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:48.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:48.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:48.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:48.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:48.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:48.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:48.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:48.206 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:48.206 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:48.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494687 2021-05-12T12:36:48.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494687 '!=' null 2021-05-12T12:36:48.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:48.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:48.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:48.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:48.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:48.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:48.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:48.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:48.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:48.795 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:48.795 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:49.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:49.154 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:49.155 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:49.155 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:49.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:49.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:49.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:49.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:49.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:49.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:49.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:49.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:49.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:50.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115707 2021-05-12T12:36:50.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115707 '!=' null 2021-05-12T12:36:50.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:50.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:50.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:50.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:50.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:50.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:50.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:50.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:50.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:50.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:50.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:50.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:50.583 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:50.583 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:50.584 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:51.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:51.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:51.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:51.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:51.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:51.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:51.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:51.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:51.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:51.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242008 2021-05-12T12:36:51.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242008 '!=' null 2021-05-12T12:36:51.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:51.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:51.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:51.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:51.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:51.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:51.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:51.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:51.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:51.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:51.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:52.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:52.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:52.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:52.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:52.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:52.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:52.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:52.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:52.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:52.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:52.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:52.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:52.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:52.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242008 2021-05-12T12:36:52.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242008 '!=' null 2021-05-12T12:36:52.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:52.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:52.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:53.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:53.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:53.100 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:53.100 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:53.100 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:53.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:53.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:53.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:53.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:53.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:53.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:53.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:54.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:54.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:54.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:54.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:54.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:54.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:54.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:54.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:54.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:54.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115690 2021-05-12T12:36:54.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115690 '!=' null 2021-05-12T12:36:54.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:54.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:54.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:54.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:54.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:54.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:54.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:54.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:54.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:54.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:54.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:54.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:54.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:54.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:54.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:55.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:55.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:55.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:55.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:55.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:55.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:55.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:55.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:55.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:55.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494673 2021-05-12T12:36:55.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494673 '!=' null 2021-05-12T12:36:55.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:55.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:55.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:56.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:56.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:56.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:56.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:56.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:56.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:56.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:56.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:56.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:56.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:56.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:56.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:56.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:56.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:56.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:56.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:56.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:56.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:56.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:56.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:56.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:57.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494636 2021-05-12T12:36:57.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494636 '!=' null 2021-05-12T12:36:57.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:57.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:57.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:57.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:57.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:57.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:57.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:57.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:57.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:57.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:57.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:57.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:57.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:57.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:57.849 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:58.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:58.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:58.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:58.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:58.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:58.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:58.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:58.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:58.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:58.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494636 2021-05-12T12:36:58.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494636 '!=' null 2021-05-12T12:36:58.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:58.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:58.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:58.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:58.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:58.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:58.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:58.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:58.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:58.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:58.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:59.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:59.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:59.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:59.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:59.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:59.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:59.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:59.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:59.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:59.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:59.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:59.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:59.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:00.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115548 2021-05-12T12:37:00.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115548 '!=' null 2021-05-12T12:37:00.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:00.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:00.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:00.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:00.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:00.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:00.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:00.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:00.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:00.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:00.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:00.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:00.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:00.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:00.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:01.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:01.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:01.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:01.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:01.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:01.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:01.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:01.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:01.244 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:01.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494598 2021-05-12T12:37:01.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494598 '!=' null 2021-05-12T12:37:01.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:01.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:01.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:01.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:01.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:01.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:01.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:01.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:01.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:01.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:01.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:02.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:02.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:02.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:02.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:02.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:02.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:02.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:02.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:02.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:02.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:02.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:02.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:02.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:03.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494598 2021-05-12T12:37:03.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494598 '!=' null 2021-05-12T12:37:03.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:03.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:03.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:03.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:03.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:03.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:03.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:03.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:03.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:03.267 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:03.267 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:03.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:03.631 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:03.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:03.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:04.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:04.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:04.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:04.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:04.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:04.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:04.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:04.121 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:04.121 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:04.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241929 2021-05-12T12:37:04.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241929 '!=' null 2021-05-12T12:37:04.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:04.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:04.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:04.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:04.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:04.708 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:04.709 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:04.709 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:04.709 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:04.709 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:04.709 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:05.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:05.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:05.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:05.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:05.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:05.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:05.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:05.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:05.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:05.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:05.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:05.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:05.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:06.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368270 2021-05-12T12:37:06.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368270 '!=' null 2021-05-12T12:37:06.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:06.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:06.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:06.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:06.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:06.168 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:06.168 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:06.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:06.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:06.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:06.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:06.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:06.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:06.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:06.535 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:07.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:07.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:07.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:07.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:07.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:07.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:07.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:07.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:07.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:07.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368269 2021-05-12T12:37:07.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368269 '!=' null 2021-05-12T12:37:07.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:07.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:07.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:07.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:07.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:07.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:07.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:07.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:07.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:07.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:07.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:07.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:07.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:07.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:07.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:08.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:08.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:08.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:08.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:08.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:08.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:08.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:08.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:08.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:08.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368269 2021-05-12T12:37:08.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368269 '!=' null 2021-05-12T12:37:08.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:08.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:08.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:09.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:09.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:09.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:09.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:09.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:09.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:09.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:09.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:09.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:09.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:09.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:09.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:09.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:09.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:09.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:09.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:09.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:09.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:09.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:09.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:09.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:10.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115618 2021-05-12T12:37:10.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115618 '!=' null 2021-05-12T12:37:10.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:10.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:10.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:10.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:10.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:10.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:10.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:10.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:10.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:10.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:10.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:10.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:10.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:10.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:10.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:11.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:11.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:11.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:11.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:11.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:11.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:11.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:11.369 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:11.369 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:11.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494672 2021-05-12T12:37:11.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494672 '!=' null 2021-05-12T12:37:11.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:11.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:11.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:12.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:12.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:12.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:12.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:12.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:12.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:12.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:12.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:12.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:12.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:12.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:12.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:12.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:12.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:12.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:12.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:12.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:12.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:12.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:12.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:12.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:13.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242016 2021-05-12T12:37:13.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242016 '!=' null 2021-05-12T12:37:13.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:13.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:13.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:13.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:13.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:13.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:13.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:13.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:13.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:13.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:13.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:13.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:13.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:13.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:13.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:14.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:14.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:14.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:14.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:14.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:14.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:14.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:14.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:14.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:14.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242016 2021-05-12T12:37:14.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242016 '!=' null 2021-05-12T12:37:14.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:14.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:14.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:14.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:14.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:14.922 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:14.922 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:14.922 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:14.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:14.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:14.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:15.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:15.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:15.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:15.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:15.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:15.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:15.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:15.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:15.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:15.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:15.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:15.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:15.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:16.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494653 2021-05-12T12:37:16.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494653 '!=' null 2021-05-12T12:37:16.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:16.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:16.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:16.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:16.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:16.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:16.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:16.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:16.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:16.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:16.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:16.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:16.808 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:16.808 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:16.808 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:17.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:17.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:17.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:17.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:17.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:17.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:17.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:17.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:17.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:17.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494651 2021-05-12T12:37:17.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494651 '!=' null 2021-05-12T12:37:17.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:17.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:17.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:17.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:17.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:17.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:17.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:17.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:17.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:17.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:17.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:18.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:18.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:18.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:18.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:18.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:18.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:18.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:18.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:18.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:18.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:18.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:18.779 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:18.779 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:19.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115671 2021-05-12T12:37:19.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115671 '!=' null 2021-05-12T12:37:19.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:19.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:19.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:19.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:19.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:19.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:19.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:19.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:19.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:19.368 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:19.368 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:19.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:19.731 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:19.731 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:19.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:20.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:20.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:20.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:20.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:20.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:20.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:20.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:20.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:20.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:20.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115671 2021-05-12T12:37:20.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115671 '!=' null 2021-05-12T12:37:20.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:20.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:20.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:20.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:20.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:20.774 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:20.774 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:20.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:20.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:20.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:20.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:21.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:21.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:21.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:21.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:21.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:21.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:21.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:21.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:21.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:21.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:21.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:21.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:21.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:22.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368321 2021-05-12T12:37:22.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368321 '!=' null 2021-05-12T12:37:22.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:22.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:22.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:22.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:22.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:22.196 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:22.196 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:22.196 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:22.196 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:22.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:22.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:22.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:22.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:22.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:22.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:23.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:23.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:23.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:23.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:23.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:23.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:23.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:23.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:23.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:23.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115616 2021-05-12T12:37:23.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115616 '!=' null 2021-05-12T12:37:23.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:23.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:23.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:23.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:23.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:23.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:23.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:23.658 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:23.658 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:23.658 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:23.658 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:24.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:24.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:24.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:24.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:24.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:24.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:24.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:24.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:24.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:24.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:24.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:24.512 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:24.512 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:24.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115616 2021-05-12T12:37:24.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115616 '!=' null 2021-05-12T12:37:24.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:24.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:24.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:25.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:25.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:25.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:25.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:25.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:25.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:25.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:25.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:25.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:25.461 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:25.461 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:25.461 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:25.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:25.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:25.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:25.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:25.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:25.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:25.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:25.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:25.966 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:26.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368258 2021-05-12T12:37:26.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368258 '!=' null 2021-05-12T12:37:26.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:26.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:26.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:26.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:26.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:26.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:26.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:26.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:26.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:26.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:26.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:26.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:26.912 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:26.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:26.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:27.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:27.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:27.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:27.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:27.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:27.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:27.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:27.405 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:27.405 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:27.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368267 2021-05-12T12:37:27.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368267 '!=' null 2021-05-12T12:37:27.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:27.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:27.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:28.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:28.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:28.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:28.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:28.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:28.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:28.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:28.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:28.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:28.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:28.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:28.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:28.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:28.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:28.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:28.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:28.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:28.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:28.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:28.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:28.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:29.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115596 2021-05-12T12:37:29.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115596 '!=' null 2021-05-12T12:37:29.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:29.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:29.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:29.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:29.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:29.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:29.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:29.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:29.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:29.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:29.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:29.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:29.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:29.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:29.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:30.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:30.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:30.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:30.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:30.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:30.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:30.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:30.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:30.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:30.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115596 2021-05-12T12:37:30.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115596 '!=' null 2021-05-12T12:37:30.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:30.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:30.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:30.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:30.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:30.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:30.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:30.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:30.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:30.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:30.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:31.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:31.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:31.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:31.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:31.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:31.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:31.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:31.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:31.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:31.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:31.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:31.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:31.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:32.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494590 2021-05-12T12:37:32.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494590 '!=' null 2021-05-12T12:37:32.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:32.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:32.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:32.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:32.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:32.462 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:32.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:32.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:32.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:32.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:32.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:32.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:32.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:32.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:32.820 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:33.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:33.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:33.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:33.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:33.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:33.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:33.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:33.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:33.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:33.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115590 2021-05-12T12:37:33.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115590 '!=' null 2021-05-12T12:37:33.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:33.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:33.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:33.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:33.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:33.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:33.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:33.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:33.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:33.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:33.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:34.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:34.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:34.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:34.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:34.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:34.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:34.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:34.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:34.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:34.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:34.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:34.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:34.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:35.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368288 2021-05-12T12:37:35.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368288 '!=' null 2021-05-12T12:37:35.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:35.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:35.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:35.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:35.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:35.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:35.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:35.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:35.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:35.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:35.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:35.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:35.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:35.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:35.777 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:36.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:36.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:36.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:36.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:36.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:36.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:36.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:36.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:36.281 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:36.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368288 2021-05-12T12:37:36.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368288 '!=' null 2021-05-12T12:37:36.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:36.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:36.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:36.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:36.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:36.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:36.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:36.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:36.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:36.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:36.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:37.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:37.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:37.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:37.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:37.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:37.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:37.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:37.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:37.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:37.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:37.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:37.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:37.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:38.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368295 2021-05-12T12:37:38.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368295 '!=' null 2021-05-12T12:37:38.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:38.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:38.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:38.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:38.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:38.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:38.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:38.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:38.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:38.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:38.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:38.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:38.716 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:38.716 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:38.716 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:39.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:39.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:39.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:39.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:39.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:39.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:39.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:39.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:39.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:39.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115621 2021-05-12T12:37:39.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115621 '!=' null 2021-05-12T12:37:39.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:39.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:39.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:39.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:39.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:39.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:39.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:39.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:39.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:39.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:39.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:40.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:40.185 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:40.186 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:40.186 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:40.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:40.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:40.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:40.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:40.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:40.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:40.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:40.678 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:40.678 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:41.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115621 2021-05-12T12:37:41.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115621 '!=' null 2021-05-12T12:37:41.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:41.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:41.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:41.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:41.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:41.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:41.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:41.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:41.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:41.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:41.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:41.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:41.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:41.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:41.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:42.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:42.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:42.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:42.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:42.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:42.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:42.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:42.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:42.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:42.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368284 2021-05-12T12:37:42.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368284 '!=' null 2021-05-12T12:37:42.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:42.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:42.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:42.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:42.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:42.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:42.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:42.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:42.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:42.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:42.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:43.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:43.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:43.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:43.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:43.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:43.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:43.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:43.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:43.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:43.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:43.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:43.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:43.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:44.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115651 2021-05-12T12:37:44.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115651 '!=' null 2021-05-12T12:37:44.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:44.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:44.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:44.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:44.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:44.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:44.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:44.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:44.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:44.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:44.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:44.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:44.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:44.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:44.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:45.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:45.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:45.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:45.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:45.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:45.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:45.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:45.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:45.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:45.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494644 2021-05-12T12:37:45.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494644 '!=' null 2021-05-12T12:37:45.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:45.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:45.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:45.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:45.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:45.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:45.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:45.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:45.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:45.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:45.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:46.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:46.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:46.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:46.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:46.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:46.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:46.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:46.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:46.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:46.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:46.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:46.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:46.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:46.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494644 2021-05-12T12:37:46.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494644 '!=' null 2021-05-12T12:37:46.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:46.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:46.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:47.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:47.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:47.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:47.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:47.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:47.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:47.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:47.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:47.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:47.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:47.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:47.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:47.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:47.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:47.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:47.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:47.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:47.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:47.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:47.956 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:47.956 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:48.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494630 2021-05-12T12:37:48.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494630 '!=' null 2021-05-12T12:37:48.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:48.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:48.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:48.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:48.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:48.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:48.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:48.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:48.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:48.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:48.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:48.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:48.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:48.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:48.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:49.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:49.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:49.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:49.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:49.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:49.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:49.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:49.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:49.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:49.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115605 2021-05-12T12:37:49.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115605 '!=' null 2021-05-12T12:37:49.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:49.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:49.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:49.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:49.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:49.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:49.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:49.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:49.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:49.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:49.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:50.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:50.344 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:50.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:50.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:50.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:50.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:50.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:50.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:50.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:50.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:50.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:50.835 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:50.835 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:51.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368292 2021-05-12T12:37:51.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368292 '!=' null 2021-05-12T12:37:51.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:51.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:51.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:51.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:51.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:51.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:51.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:51.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:51.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:51.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:51.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:51.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:51.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:51.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:51.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:52.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:52.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:52.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:52.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:52.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:52.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:52.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:52.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:52.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:52.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368292 2021-05-12T12:37:52.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368292 '!=' null 2021-05-12T12:37:52.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:52.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:52.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:52.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:52.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:52.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:52.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:52.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:52.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:52.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:52.900 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:53.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:53.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:53.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:53.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:53.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:53.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:53.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:53.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:53.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:53.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:53.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:53.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:53.757 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:54.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115615 2021-05-12T12:37:54.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115615 '!=' null 2021-05-12T12:37:54.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:54.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:54.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:54.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:54.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:54.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:54.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:54.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:54.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:54.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:54.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:54.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:54.702 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:54.702 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:54.702 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:55.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:55.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:55.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:55.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:55.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:55.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:55.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:55.222 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:55.222 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:55.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368291 2021-05-12T12:37:55.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368291 '!=' null 2021-05-12T12:37:55.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:55.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:55.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:55.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:55.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:55.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:55.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:55.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:55.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:55.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:55.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:56.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:56.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:56.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:56.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:56.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:56.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:56.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:56.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:56.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:56.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:56.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:56.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:56.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:57.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368291 2021-05-12T12:37:57.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368291 '!=' null 2021-05-12T12:37:57.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:57.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:57.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:57.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:57.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:57.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:57.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:57.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:57.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:57.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:57.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:57.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:57.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:57.613 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:57.613 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:58.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:58.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:58.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:58.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:58.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:58.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:58.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:58.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:58.100 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:58.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368287 2021-05-12T12:37:58.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368287 '!=' null 2021-05-12T12:37:58.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:58.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:58.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:58.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:58.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:58.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:58.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:58.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:58.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:58.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:58.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:59.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:59.041 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:59.041 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:59.041 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:59.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:59.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:59.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:59.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:59.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:59.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:59.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:59.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:59.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:00.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115660 2021-05-12T12:38:00.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115660 '!=' null 2021-05-12T12:38:00.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:00.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:00.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:00.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:00.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:00.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:00.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:00.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:00.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:00.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:00.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:00.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:00.464 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:00.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:00.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:00.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:00.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:00.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:00.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:00.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:00.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:00.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:00.956 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:00.957 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:01.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494690 2021-05-12T12:38:01.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494690 '!=' null 2021-05-12T12:38:01.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:01.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:01.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:01.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:01.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:01.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:01.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:01.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:01.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:01.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:01.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:01.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:01.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:01.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:01.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:02.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:02.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:02.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:02.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:02.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:02.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:02.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:02.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:02.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:02.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494690 2021-05-12T12:38:02.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494690 '!=' null 2021-05-12T12:38:02.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:02.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:02.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:03.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:03.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:03.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:03.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:03.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:03.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:03.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:03.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:03.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:03.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:03.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:03.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:03.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:03.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:03.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:03.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:03.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:03.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:03.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:03.955 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:03.955 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:04.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115705 2021-05-12T12:38:04.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115705 '!=' null 2021-05-12T12:38:04.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:04.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:04.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:04.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:04.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:04.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:04.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:04.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:04.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:04.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:04.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:04.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:04.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:04.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:04.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:05.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:05.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:05.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:05.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:05.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:05.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:05.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:05.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:05.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:05.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368348 2021-05-12T12:38:05.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368348 '!=' null 2021-05-12T12:38:05.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:05.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:05.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:05.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:05.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:05.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:05.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:05.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:05.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:05.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:05.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:06.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:06.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:06.360 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:06.360 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:06.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:06.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:06.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:06.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:06.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:06.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:06.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:06.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:06.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:07.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368331 2021-05-12T12:38:07.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368331 '!=' null 2021-05-12T12:38:07.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:07.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:07.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:07.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:07.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:07.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:07.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:07.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:07.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:07.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:07.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:07.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:07.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:07.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:07.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:08.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:08.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:08.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:08.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:08.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:08.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:08.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:08.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:08.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:08.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368331 2021-05-12T12:38:08.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368331 '!=' null 2021-05-12T12:38:08.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:08.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:08.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:08.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:08.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:08.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:08.951 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:08.951 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:08.952 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:08.952 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:08.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:09.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:09.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:09.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:09.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:09.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:09.824 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:09.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:09.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:09.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:09.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:09.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:09.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:09.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:10.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115630 2021-05-12T12:38:10.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115630 '!=' null 2021-05-12T12:38:10.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:10.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:10.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:10.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:10.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:10.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:10.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:10.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:10.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:10.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:10.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:10.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:10.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:10.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:10.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:11.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:11.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:11.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:11.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:11.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:11.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:11.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:11.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:11.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:11.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368289 2021-05-12T12:38:11.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368289 '!=' null 2021-05-12T12:38:11.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:11.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:11.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:11.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:11.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:11.857 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:11.857 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:11.857 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:11.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:11.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:11.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:12.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:12.226 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:12.226 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:12.226 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:12.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:12.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:12.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:12.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:12.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:12.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:12.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:12.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:12.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:13.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368289 2021-05-12T12:38:13.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368289 '!=' null 2021-05-12T12:38:13.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:13.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:13.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:13.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:13.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:13.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:13.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:13.313 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:13.313 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:13.313 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:13.313 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:13.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:13.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:13.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:13.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:14.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:14.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:14.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:14.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:14.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:14.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:14.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:14.155 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:14.155 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:14.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115606 2021-05-12T12:38:14.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115606 '!=' null 2021-05-12T12:38:14.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:14.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:14.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:14.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:14.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:14.805 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:14.805 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:14.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:14.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:14.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:14.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:15.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:15.160 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:15.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:15.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:15.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:15.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:15.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:15.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:15.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:15.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:15.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:15.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:15.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:16.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494642 2021-05-12T12:38:16.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494642 '!=' null 2021-05-12T12:38:16.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:16.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:16.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:16.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:16.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:16.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:16.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:16.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:16.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:16.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:16.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:16.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:16.590 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:16.590 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:16.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:17.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:17.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:17.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:17.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:17.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:17.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:17.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:17.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:17.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:17.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494630 2021-05-12T12:38:17.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494630 '!=' null 2021-05-12T12:38:17.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:17.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:17.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:17.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:17.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:17.678 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:17.678 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:17.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:17.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:17.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:17.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:18.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:18.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:18.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:18.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:18.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:18.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:18.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:18.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:18.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:18.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:18.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:18.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:18.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:18.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494630 2021-05-12T12:38:18.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494630 '!=' null 2021-05-12T12:38:18.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:18.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:18.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:19.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:19.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:19.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:19.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:19.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:19.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:19.100 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:19.100 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:19.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:19.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:19.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:19.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:19.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:19.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:19.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:19.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:19.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:19.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:19.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:19.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:19.948 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:20.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115574 2021-05-12T12:38:20.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115574 '!=' null 2021-05-12T12:38:20.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:20.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:20.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:20.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:20.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:20.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:20.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:20.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:20.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:20.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:20.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:20.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:20.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:20.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:20.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:21.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:21.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:21.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:21.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:21.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:21.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:21.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:21.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:21.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:21.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368264 2021-05-12T12:38:21.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368264 '!=' null 2021-05-12T12:38:21.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:21.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:21.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:22.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:22.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:22.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:22.031 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:22.031 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:22.031 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:22.031 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:22.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:22.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:22.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:22.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:22.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:22.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:22.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:22.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:22.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:22.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:22.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:22.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:22.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:22.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:23.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115586 2021-05-12T12:38:23.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115586 '!=' null 2021-05-12T12:38:23.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:23.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:23.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:23.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:23.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:23.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:23.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:23.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:23.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:23.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:23.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:23.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:23.846 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:23.847 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:23.847 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:24.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:24.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:24.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:24.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:24.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:24.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:24.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:24.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:24.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:24.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115586 2021-05-12T12:38:24.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115586 '!=' null 2021-05-12T12:38:24.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:24.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:24.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:24.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:24.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:24.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:24.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:24.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:24.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:24.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:24.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:25.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:25.300 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:25.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:25.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:25.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:25.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:25.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:25.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:25.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:25.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:25.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:25.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:25.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:26.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368265 2021-05-12T12:38:26.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368265 '!=' null 2021-05-12T12:38:26.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:26.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:26.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:26.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:26.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:26.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:26.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:26.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:26.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:26.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:26.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:26.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:26.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:26.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:26.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:27.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:27.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:27.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:27.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:27.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:27.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:27.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:27.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:27.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:27.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368239 2021-05-12T12:38:27.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368239 '!=' null 2021-05-12T12:38:27.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:27.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:27.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:27.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:27.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:27.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:27.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:27.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:27.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:27.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:27.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:28.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:28.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:28.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:28.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:28.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:28.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:28.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:28.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:28.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:28.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:28.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:28.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:28.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:29.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368239 2021-05-12T12:38:29.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368239 '!=' null 2021-05-12T12:38:29.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:29.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:29.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:29.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:29.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:29.308 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:29.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:29.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:29.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:29.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:29.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:29.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:29.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:29.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:29.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:30.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:30.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:30.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:30.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:30.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:30.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:30.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:30.185 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:30.185 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:30.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115575 2021-05-12T12:38:30.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115575 '!=' null 2021-05-12T12:38:30.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:30.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:30.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:30.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:30.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:30.772 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:30.772 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:30.772 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:30.773 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:30.773 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:30.773 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:31.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:31.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:31.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:31.133 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:31.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:31.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:31.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:31.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:31.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:31.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:31.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:31.626 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:31.626 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:32.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494648 2021-05-12T12:38:32.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494648 '!=' null 2021-05-12T12:38:32.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:32.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:32.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:32.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:32.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:32.211 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:32.211 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:32.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:32.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:32.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:32.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:32.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:32.569 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:32.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:32.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:33.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:33.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:33.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:33.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:33.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:33.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:33.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:33.061 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:33.061 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:33.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115681 2021-05-12T12:38:33.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115681 '!=' null 2021-05-12T12:38:33.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:33.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:33.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:33.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:33.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:33.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:33.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:33.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:33.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:33.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:33.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:34.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:34.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:34.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:34.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:34.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:34.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:34.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:34.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:34.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:34.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:34.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:34.491 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:34.491 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:34.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115681 2021-05-12T12:38:34.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115681 '!=' null 2021-05-12T12:38:34.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:34.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:34.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:35.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:35.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:35.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:35.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:35.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:35.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:35.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:35.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:35.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:35.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:35.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:35.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:36.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:36.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:36.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:36.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:36.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:36.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:36.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:36.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:36.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:36.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368333 2021-05-12T12:38:36.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368333 '!=' null 2021-05-12T12:38:36.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:36.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:36.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:36.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:36.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:36.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:36.600 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:36.600 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:36.600 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:36.600 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:36.601 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:36.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:36.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:36.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:36.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:37.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:37.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:37.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:37.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:37.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:37.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:37.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:37.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:37.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:37.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368295 2021-05-12T12:38:37.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368295 '!=' null 2021-05-12T12:38:37.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:37.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:37.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:38.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:38.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:38.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:38.046 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:38.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:38.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:38.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:38.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:38.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:38.404 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:38.405 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:38.405 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:38.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:38.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:38.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:38.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:38.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:38.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:38.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:38.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:38.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:39.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115570 2021-05-12T12:38:39.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115570 '!=' null 2021-05-12T12:38:39.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:39.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:39.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:39.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:39.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:39.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:39.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:39.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:39.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:39.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:39.589 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:39.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:39.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:39.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:39.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:40.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:40.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:40.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:40.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:40.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:40.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:40.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:40.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:40.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:40.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115570 2021-05-12T12:38:40.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115570 '!=' null 2021-05-12T12:38:40.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:40.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:40.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:41.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:41.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:41.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:41.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:41.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:41.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:41.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:41.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:41.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:41.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:41.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:41.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:41.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:41.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:41.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:41.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:41.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:41.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:41.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:41.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:41.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:42.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368259 2021-05-12T12:38:42.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368259 '!=' null 2021-05-12T12:38:42.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:42.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:42.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:42.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:42.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:42.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:42.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:42.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:42.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:42.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:42.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:42.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:42.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:42.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:42.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:43.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:43.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:43.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:43.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:43.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:43.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:43.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:43.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:43.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:43.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115509 2021-05-12T12:38:43.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115509 '!=' null 2021-05-12T12:38:43.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:43.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:43.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:43.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:43.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:43.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:43.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:43.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:43.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:43.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:43.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:44.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:44.311 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:44.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:44.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:44.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:44.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:44.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:44.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:44.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:44.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:44.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:44.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:44.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:45.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494546 2021-05-12T12:38:45.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494546 '!=' null 2021-05-12T12:38:45.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:45.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:45.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:45.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:45.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:45.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:45.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:45.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:45.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:45.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:45.421 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:45.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:45.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:45.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:45.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:46.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:46.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:46.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:46.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:46.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:46.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:46.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:46.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:46.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:46.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494546 2021-05-12T12:38:46.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494546 '!=' null 2021-05-12T12:38:46.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:46.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:46.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:46.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:46.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:46.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:46.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:46.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:46.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:46.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:46.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:47.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:47.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:47.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:47.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:47.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:47.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:47.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:47.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:47.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:47.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:47.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:47.711 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:47.711 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:48.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494556 2021-05-12T12:38:48.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494556 '!=' null 2021-05-12T12:38:48.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:48.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:48.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:48.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:48.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:48.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:48.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:48.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:48.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:48.346 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:48.346 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:48.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:48.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:48.696 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:48.696 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:49.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:49.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:49.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:49.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:49.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:49.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:49.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:49.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:49.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:49.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115536 2021-05-12T12:38:49.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115536 '!=' null 2021-05-12T12:38:49.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:49.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:49.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:49.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:49.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:49.814 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:49.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:49.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:49.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:49.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:49.816 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:50.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:50.172 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:50.172 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:50.173 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:50.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:50.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:50.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:50.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:50.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:50.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:50.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:50.663 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:50.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:51.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115536 2021-05-12T12:38:51.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115536 '!=' null 2021-05-12T12:38:51.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:51.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:51.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:51.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:51.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:51.261 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:51.261 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:51.261 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:51.262 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:51.262 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:51.262 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:51.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:51.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:51.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:51.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:52.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:52.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:52.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:52.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:52.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:52.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:52.106 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:52.107 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:52.107 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:52.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368276 2021-05-12T12:38:52.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368276 '!=' null 2021-05-12T12:38:52.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:52.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:52.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:52.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:52.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:52.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:52.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:52.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:52.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:52.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:52.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:53.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:53.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:53.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:53.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:53.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:53.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:53.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:53.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:53.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:53.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:53.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:53.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:53.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:54.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115555 2021-05-12T12:38:54.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115555 '!=' null 2021-05-12T12:38:54.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:54.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:54.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:54.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:54.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:54.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:54.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:54.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:54.204 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:54.204 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:54.204 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:54.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:54.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:54.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:54.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:55.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:55.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:55.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:55.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:55.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:55.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:55.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:55.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:55.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:55.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368286 2021-05-12T12:38:55.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368286 '!=' null 2021-05-12T12:38:55.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:55.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:55.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:55.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:55.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:55.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:55.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:55.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:55.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:55.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:55.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:56.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:56.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:56.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:56.026 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:56.516 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:56.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:56.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:56.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:56.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:56.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:56.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:56.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:56.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:57.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368286 2021-05-12T12:38:57.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368286 '!=' null 2021-05-12T12:38:57.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:57.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:57.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:57.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:57.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:57.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:57.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:57.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:57.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:57.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:57.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:57.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:57.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:57.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:57.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:57.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:57.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:57.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:57.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:57.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:57.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:57.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:57.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:57.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:58.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368333 2021-05-12T12:38:58.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368333 '!=' null 2021-05-12T12:38:58.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:58.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:58.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:58.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:58.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:58.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:58.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:58.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:58.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:58.546 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:58.546 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:58.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:58.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:58.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:58.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:59.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:59.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:59.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:59.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:59.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:59.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:59.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:59.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:59.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:59.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115669 2021-05-12T12:38:59.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115669 '!=' null 2021-05-12T12:38:59.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:59.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:59.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:00.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:00.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:00.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:00.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:00.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:00.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:00.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:00.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:00.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:00.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:00.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:00.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:00.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:00.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:00.845 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:00.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:00.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:00.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:00.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:00.847 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:00.847 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:01.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115669 2021-05-12T12:39:01.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115669 '!=' null 2021-05-12T12:39:01.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:01.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:01.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:01.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:01.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:01.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:01.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:01.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:01.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:01.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:01.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:01.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:01.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:01.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:01.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:02.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:02.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:02.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:02.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:02.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:02.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:02.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:02.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:02.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:02.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494642 2021-05-12T12:39:02.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494642 '!=' null 2021-05-12T12:39:02.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:02.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:02.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:02.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:02.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:02.957 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:02.957 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:02.957 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:02.958 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:02.958 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:02.958 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:03.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:03.328 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:03.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:03.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:03.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:03.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:03.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:03.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:03.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:03.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:03.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:03.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:03.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:04.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115634 2021-05-12T12:39:04.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115634 '!=' null 2021-05-12T12:39:04.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:04.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:04.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:04.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:04.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:04.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:04.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:04.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:04.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:04.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:04.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:04.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:04.805 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:04.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:04.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:05.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:05.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:05.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:05.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:05.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:05.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:05.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:05.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:05.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:05.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368281 2021-05-12T12:39:05.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368281 '!=' null 2021-05-12T12:39:05.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:05.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:05.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:05.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:05.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:05.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:05.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:05.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:05.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:05.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:05.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:06.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:06.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:06.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:06.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:06.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:06.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:06.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:06.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:06.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:06.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:06.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:06.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:06.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:07.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368238 2021-05-12T12:39:07.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368238 '!=' null 2021-05-12T12:39:07.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:07.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:07.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:07.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:07.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:07.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:07.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:07.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:07.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:07.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:07.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:07.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:07.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:07.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:07.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:08.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:08.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:08.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:08.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:08.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:08.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:08.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:08.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:08.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:08.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368238 2021-05-12T12:39:08.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368238 '!=' null 2021-05-12T12:39:08.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:08.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:08.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:08.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:08.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:08.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:08.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:08.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:08.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:08.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:08.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:09.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:09.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:09.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:09.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:09.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:09.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:09.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:09.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:09.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:09.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:09.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:09.850 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:09.850 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:10.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115538 2021-05-12T12:39:10.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115538 '!=' null 2021-05-12T12:39:10.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:10.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:10.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:10.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:10.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:10.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:10.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:10.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:10.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:10.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:10.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:10.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:10.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:10.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:10.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:11.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:11.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:11.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:11.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:11.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:11.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:11.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:11.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:11.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:11.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368235 2021-05-12T12:39:11.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368235 '!=' null 2021-05-12T12:39:11.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:11.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:11.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:11.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:11.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:11.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:11.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:11.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:11.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:11.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:11.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:12.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:12.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:12.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:12.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:12.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:12.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:12.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:12.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:12.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:12.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:12.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:12.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:12.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:13.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368235 2021-05-12T12:39:13.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368235 '!=' null 2021-05-12T12:39:13.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:13.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:13.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:13.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:13.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:13.325 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:13.325 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:13.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:13.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:13.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:13.327 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:13.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:13.687 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:13.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:13.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:14.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:14.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:14.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:14.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:14.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:14.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:14.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:14.179 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:14.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:14.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115529 2021-05-12T12:39:14.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115529 '!=' null 2021-05-12T12:39:14.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:14.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:14.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:14.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:14.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:14.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:14.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:14.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:14.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:14.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:14.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:15.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:15.129 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:15.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:15.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:15.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:15.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:15.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:15.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:15.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:15.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:15.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:15.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:15.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:16.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494596 2021-05-12T12:39:16.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494596 '!=' null 2021-05-12T12:39:16.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:16.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:16.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:16.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:16.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:16.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:16.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:16.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:16.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:16.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:16.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:16.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:16.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:16.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:16.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:17.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:17.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:17.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:17.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:17.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:17.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:17.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:17.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:17.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:17.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494635 2021-05-12T12:39:17.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494635 '!=' null 2021-05-12T12:39:17.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:17.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:17.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:17.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:17.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:17.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:17.665 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:17.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:17.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:17.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:17.667 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:18.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:18.017 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:18.017 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:18.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:18.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:18.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:18.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:18.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:18.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:18.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:18.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:18.506 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:18.507 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:19.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494635 2021-05-12T12:39:19.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494635 '!=' null 2021-05-12T12:39:19.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:19.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:19.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:19.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:19.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:19.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:19.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:19.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:19.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:19.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:19.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:19.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:19.474 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:19.475 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:19.475 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:19.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:19.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:19.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:19.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:19.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:19.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:19.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:19.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:19.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:20.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115609 2021-05-12T12:39:20.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115609 '!=' null 2021-05-12T12:39:20.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:20.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:20.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:20.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:20.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:20.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:20.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:20.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:20.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:20.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:20.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:20.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:20.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:20.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:20.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:21.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:21.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:21.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:21.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:21.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:21.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:21.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:21.412 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:21.412 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:21.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368268 2021-05-12T12:39:21.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368268 '!=' null 2021-05-12T12:39:21.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:21.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:21.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:22.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:22.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:22.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:22.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:22.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:22.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:22.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:22.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:22.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:22.356 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:22.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:22.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:22.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:22.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:22.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:22.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:22.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:22.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:22.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:22.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:22.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:23.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115584 2021-05-12T12:39:23.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115584 '!=' null 2021-05-12T12:39:23.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:23.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:23.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:23.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:23.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:23.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:23.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:23.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:23.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:23.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:23.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:23.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:23.804 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:23.804 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:23.804 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:24.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:24.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:24.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:24.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:24.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:24.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:24.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:24.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:24.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:24.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115584 2021-05-12T12:39:24.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115584 '!=' null 2021-05-12T12:39:24.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:24.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:24.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:24.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:24.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:24.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:24.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:24.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:24.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:24.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:24.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:25.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:25.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:25.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:25.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:25.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:25.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:25.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:25.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:25.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:25.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:25.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:25.736 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:25.736 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:26.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368293 2021-05-12T12:39:26.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368293 '!=' null 2021-05-12T12:39:26.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:26.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:26.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:26.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:26.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:26.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:26.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:26.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:26.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:26.325 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:26.325 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:26.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:26.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:26.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:26.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:27.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:27.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:27.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:27.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:27.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:27.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:27.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:27.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:27.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:27.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368288 2021-05-12T12:39:27.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368288 '!=' null 2021-05-12T12:39:27.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:27.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:27.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:27.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:27.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:27.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:27.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:27.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:27.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:27.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:27.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:28.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:28.125 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:28.125 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:28.125 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:28.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:28.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:28.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:28.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:28.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:28.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:28.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:28.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:28.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:29.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368288 2021-05-12T12:39:29.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368288 '!=' null 2021-05-12T12:39:29.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:29.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:29.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:29.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:29.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:29.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:29.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:29.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:29.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:29.225 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:29.226 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:29.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:29.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:29.596 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:29.596 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:30.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:30.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:30.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:30.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:30.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:30.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:30.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:30.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:30.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:30.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115614 2021-05-12T12:39:30.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115614 '!=' null 2021-05-12T12:39:30.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:30.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:30.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:30.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:30.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:30.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:30.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:30.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:30.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:30.674 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:30.674 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:31.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:31.043 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:31.043 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:31.044 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:31.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:31.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:31.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:31.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:31.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:31.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:31.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:31.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:31.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:32.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494672 2021-05-12T12:39:32.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494672 '!=' null 2021-05-12T12:39:32.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:32.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:32.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:32.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:32.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:32.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:32.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:32.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:32.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:32.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:32.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:32.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:32.493 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:32.493 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:32.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:32.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:32.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:32.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:32.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:32.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:32.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:32.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:32.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:32.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:33.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115698 2021-05-12T12:39:33.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115698 '!=' null 2021-05-12T12:39:33.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:33.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:33.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:33.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:33.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:33.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:33.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:33.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:33.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:33.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:33.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:33.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:33.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:33.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:33.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:34.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:34.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:34.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:34.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:34.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:34.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:34.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:34.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:34.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:34.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115698 2021-05-12T12:39:34.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115698 '!=' null 2021-05-12T12:39:34.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:34.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:34.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:35.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:35.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:35.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:35.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:35.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:35.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:35.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:35.026 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:35.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:35.380 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:35.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:35.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:35.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:35.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:35.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:35.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:35.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:35.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:35.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:35.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:35.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:36.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368341 2021-05-12T12:39:36.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368341 '!=' null 2021-05-12T12:39:36.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:36.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:36.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:36.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:36.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:36.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:36.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:36.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:36.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:36.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:36.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:36.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:36.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:36.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:36.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:37.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:37.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:37.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:37.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:37.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:37.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:37.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:37.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:37.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:37.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368342 2021-05-12T12:39:37.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368342 '!=' null 2021-05-12T12:39:37.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:37.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:37.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:37.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:37.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:37.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:37.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:37.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:37.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:37.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:37.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:38.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:38.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:38.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:38.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:38.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:38.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:38.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:38.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:38.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:38.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:38.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:38.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:38.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:39.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115650 2021-05-12T12:39:39.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115650 '!=' null 2021-05-12T12:39:39.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:39.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:39.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:39.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:39.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:39.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:39.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:39.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:39.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:39.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:39.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:39.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:39.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:39.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:39.754 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:40.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:40.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:40.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:40.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:40.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:40.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:40.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:40.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:40.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:40.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115650 2021-05-12T12:39:40.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115650 '!=' null 2021-05-12T12:39:40.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:40.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:40.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:40.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:40.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:40.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:40.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:40.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:40.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:40.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:40.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:41.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:41.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:41.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:41.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:41.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:41.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:41.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:41.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:41.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:41.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:41.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:41.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:41.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:42.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368320 2021-05-12T12:39:42.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368320 '!=' null 2021-05-12T12:39:42.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:42.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:42.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:42.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:42.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:42.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:42.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:42.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:42.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:42.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:42.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:42.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:42.644 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:42.644 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:42.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:43.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:43.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:43.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:43.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:43.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:43.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:43.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:43.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:43.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:43.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115617 2021-05-12T12:39:43.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115617 '!=' null 2021-05-12T12:39:43.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:43.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:43.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:43.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:43.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:43.744 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:43.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:43.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:43.746 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:43.746 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:43.746 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:44.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:44.125 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:44.125 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:44.126 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:44.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:44.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:44.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:44.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:44.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:44.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:44.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:44.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:44.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:45.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115617 2021-05-12T12:39:45.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115617 '!=' null 2021-05-12T12:39:45.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:45.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:45.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:45.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:45.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:45.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:45.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:45.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:45.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:45.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:45.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:45.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:45.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:45.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:45.557 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:46.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:46.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:46.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:46.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:46.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:46.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:46.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:46.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:46.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:46.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494609 2021-05-12T12:39:46.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494609 '!=' null 2021-05-12T12:39:46.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:46.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:46.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:46.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:46.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:46.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:46.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:46.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:46.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:46.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:46.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:47.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:47.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:47.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:47.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:47.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:47.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:47.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:47.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:47.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:47.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:47.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:47.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:47.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:47.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494620 2021-05-12T12:39:47.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494620 '!=' null 2021-05-12T12:39:47.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:47.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:47.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:48.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:48.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:48.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:48.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:48.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:48.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:48.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:48.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:48.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:48.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:48.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:48.457 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:48.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:48.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:48.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:48.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:48.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:48.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:48.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:48.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:48.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:49.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115552 2021-05-12T12:39:49.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115552 '!=' null 2021-05-12T12:39:49.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:49.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:49.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:49.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:49.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:49.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:49.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:49.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:49.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:49.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:49.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:49.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:49.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:49.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:49.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:50.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:50.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:50.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:50.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:50.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:50.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:50.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:50.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:50.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:50.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115552 2021-05-12T12:39:50.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115552 '!=' null 2021-05-12T12:39:50.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:50.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:50.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:50.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:50.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:50.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:50.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:50.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:50.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:50.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:50.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:51.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:51.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:51.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:51.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:51.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:51.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:51.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:51.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:51.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:51.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:51.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:51.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:51.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:52.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368263 2021-05-12T12:39:52.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368263 '!=' null 2021-05-12T12:39:52.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:52.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:52.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:52.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:52.434 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:52.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:52.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:52.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:52.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:52.436 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:52.436 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:52.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:52.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:52.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:52.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:53.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:53.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:53.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:53.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:53.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:53.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:53.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:53.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:53.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:53.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115518 2021-05-12T12:39:53.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115518 '!=' null 2021-05-12T12:39:53.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:53.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:53.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:53.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:53.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:53.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:53.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:53.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:53.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:53.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:53.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:54.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:54.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:54.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:54.304 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:54.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:54.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:54.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:54.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:54.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:54.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:54.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:54.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:54.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:55.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368210 2021-05-12T12:39:55.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368210 '!=' null 2021-05-12T12:39:55.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:55.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:55.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:55.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:55.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:55.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:55.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:55.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:55.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:55.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:55.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:55.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:55.763 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:55.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:55.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:56.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:56.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:56.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:56.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:56.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:56.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:56.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:56.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:56.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:56.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368210 2021-05-12T12:39:56.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368210 '!=' null 2021-05-12T12:39:56.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:56.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:56.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:56.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:56.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:56.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:56.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:56.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:56.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:56.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:56.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:57.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:57.202 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:57.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:57.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:57.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:57.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:57.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:57.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:57.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:57.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:57.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:57.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:57.694 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:58.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368252 2021-05-12T12:39:58.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368252 '!=' null 2021-05-12T12:39:58.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:58.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:58.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:58.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:58.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:58.290 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:58.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:58.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:58.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:58.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:58.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:58.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:58.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:58.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:58.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:59.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:59.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:59.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:59.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:59.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:59.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:59.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:59.145 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:59.146 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:59.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115540 2021-05-12T12:39:59.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115540 '!=' null 2021-05-12T12:39:59.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:59.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:59.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:59.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:59.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:59.734 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:59.735 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:59.735 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:59.735 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:59.735 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:59.736 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:00.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:00.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:00.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:00.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:00.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:00.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:00.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:00.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:00.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:00.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:00.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:00.573 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:00.573 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:01.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115540 2021-05-12T12:40:01.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115540 '!=' null 2021-05-12T12:40:01.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:01.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:01.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:01.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:01.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:01.151 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:01.151 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:01.151 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:01.151 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:01.152 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:01.152 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:01.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:01.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:01.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:01.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:02.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:02.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:02.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:02.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:02.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:02.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:02.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:02.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:02.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:02.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494625 2021-05-12T12:40:02.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494625 '!=' null 2021-05-12T12:40:02.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:02.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:02.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:02.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:02.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:02.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:02.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:02.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:02.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:02.650 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:02.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:03.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:03.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:03.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:03.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:03.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:03.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:03.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:03.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:03.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:03.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:03.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:03.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:03.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:03.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241938 2021-05-12T12:40:03.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241938 '!=' null 2021-05-12T12:40:03.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:03.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:03.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:04.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:04.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:04.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:04.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:04.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:04.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:04.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:04.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:04.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:04.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:04.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:04.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:04.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:04.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:04.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:04.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:04.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:04.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:04.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:04.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:04.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:05.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494649 2021-05-12T12:40:05.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494649 '!=' null 2021-05-12T12:40:05.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:05.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:05.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:05.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:05.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:05.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:05.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:05.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:05.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:05.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:05.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:05.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:05.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:05.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:05.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:06.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:06.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:06.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:06.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:06.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:06.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:06.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:06.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:06.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:06.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494649 2021-05-12T12:40:06.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494649 '!=' null 2021-05-12T12:40:06.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:06.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:06.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:06.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:06.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:06.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:06.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:06.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:06.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:06.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:06.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:07.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:07.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:07.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:07.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:07.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:07.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:07.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:07.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:07.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:07.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:07.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:07.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:07.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:08.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494682 2021-05-12T12:40:08.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494682 '!=' null 2021-05-12T12:40:08.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:08.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:08.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:08.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:08.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:08.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:08.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:08.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:08.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:08.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:08.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:08.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:08.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:08.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:08.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:09.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:09.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:09.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:09.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:09.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:09.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:09.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:09.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:09.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:09.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115657 2021-05-12T12:40:09.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115657 '!=' null 2021-05-12T12:40:09.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:09.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:09.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:10.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:10.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:10.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:10.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:10.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:10.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:10.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:10.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:10.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:10.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:10.384 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:10.384 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:10.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:10.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:10.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:10.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:10.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:10.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:10.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:10.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:10.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:11.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368299 2021-05-12T12:40:11.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368299 '!=' null 2021-05-12T12:40:11.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:11.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:11.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:11.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:11.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:11.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:11.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:11.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:11.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:11.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:11.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:11.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:11.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:11.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:11.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:12.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:12.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:12.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:12.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:12.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:12.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:12.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:12.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:12.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:12.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368299 2021-05-12T12:40:12.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368299 '!=' null 2021-05-12T12:40:12.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:12.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:12.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:12.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:12.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:12.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:12.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:12.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:12.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:12.981 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:12.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:13.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:13.342 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:13.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:13.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:13.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:13.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:13.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:13.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:13.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:13.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:13.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:13.840 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:13.840 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:14.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241935 2021-05-12T12:40:14.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241935 '!=' null 2021-05-12T12:40:14.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:14.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:14.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:14.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:14.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:14.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:14.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:14.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:14.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:14.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:14.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:14.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:14.798 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:14.798 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:14.799 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:15.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:15.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:15.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:15.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:15.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:15.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:15.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:15.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:15.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:15.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494600 2021-05-12T12:40:15.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494600 '!=' null 2021-05-12T12:40:15.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:15.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:15.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:15.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:15.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:15.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:15.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:15.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:15.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:15.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:15.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:16.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:16.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:16.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:16.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:16.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:16.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:16.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:16.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:16.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:16.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:16.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:16.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:16.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:17.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494600 2021-05-12T12:40:17.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494600 '!=' null 2021-05-12T12:40:17.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:17.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:17.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:17.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:17.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:17.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:17.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:17.322 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:17.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:17.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:17.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:17.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:17.684 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:17.684 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:17.685 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:18.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:18.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:18.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:18.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:18.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:18.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:18.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:18.192 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:18.192 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:18.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494584 2021-05-12T12:40:18.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494584 '!=' null 2021-05-12T12:40:18.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:18.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:18.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:18.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:18.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:18.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:18.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:18.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:18.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:18.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:18.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:19.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:19.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:19.176 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:19.176 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:19.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:19.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:19.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:19.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:19.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:19.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:19.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:19.661 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:19.661 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:20.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115496 2021-05-12T12:40:20.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115496 '!=' null 2021-05-12T12:40:20.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:20.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:20.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:20.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:20.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:20.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:20.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:20.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:20.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:20.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:20.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:20.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:20.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:20.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:20.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:21.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:21.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:21.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:21.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:21.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:21.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:21.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:21.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:21.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:21.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494587 2021-05-12T12:40:21.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494587 '!=' null 2021-05-12T12:40:21.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:21.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:21.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:21.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:21.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:21.710 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:21.710 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:21.711 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:21.711 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:21.711 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:21.711 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:22.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:22.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:22.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:22.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:22.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:22.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:22.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:22.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:22.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:22.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:22.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:22.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:22.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:23.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494587 2021-05-12T12:40:23.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494587 '!=' null 2021-05-12T12:40:23.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:23.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:23.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:23.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:23.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:23.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:23.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:23.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:23.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:23.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:23.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:23.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:23.505 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:23.506 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:23.506 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:23.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:23.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:23.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:23.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:23.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:23.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:23.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:23.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:23.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:24.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115577 2021-05-12T12:40:24.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115577 '!=' null 2021-05-12T12:40:24.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:24.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:24.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:24.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:24.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:24.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:24.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:24.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:24.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:24.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:24.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:24.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:25.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:25.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:25.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:25.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:25.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:25.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:25.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:25.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:25.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:25.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:25.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:25.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:25.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241936 2021-05-12T12:40:25.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241936 '!=' null 2021-05-12T12:40:25.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:25.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:25.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:26.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:26.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:26.061 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:26.062 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:26.062 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:26.062 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:26.062 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:26.063 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:26.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:26.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:26.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:26.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:26.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:26.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:26.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:26.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:26.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:26.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:26.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:26.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:26.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:27.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241970 2021-05-12T12:40:27.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241970 '!=' null 2021-05-12T12:40:27.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:27.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:27.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:27.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:27.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:27.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:27.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:27.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:27.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:27.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:27.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:27.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:27.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:27.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:27.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:28.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:28.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:28.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:28.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:28.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:28.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:28.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:28.397 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:28.397 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:28.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241970 2021-05-12T12:40:28.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241970 '!=' null 2021-05-12T12:40:28.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:28.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:28.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:29.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:29.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:29.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:29.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:29.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:29.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:29.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:29.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:29.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:29.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:29.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:29.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:29.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:29.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:29.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:29.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:29.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:29.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:29.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:29.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:29.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:30.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862932 2021-05-12T12:40:30.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862932 '!=' null 2021-05-12T12:40:30.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:30.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:30.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:30.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:30.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:30.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:30.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:30.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:30.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:30.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:30.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:30.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:30.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:30.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:30.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:31.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:31.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:31.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:31.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:31.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:31.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:31.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:31.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:31.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:31.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241932 2021-05-12T12:40:31.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241932 '!=' null 2021-05-12T12:40:31.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:31.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:31.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:32.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:32.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:32.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:32.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:32.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:32.028 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:32.028 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:32.028 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:32.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:32.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:32.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:32.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:32.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:32.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:32.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:32.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:32.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:32.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:32.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:32.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:32.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:33.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862790 2021-05-12T12:40:33.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862790 '!=' null 2021-05-12T12:40:33.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:33.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:33.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:33.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:33.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:33.505 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:33.505 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:33.505 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:33.506 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:33.506 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:33.506 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:33.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:33.871 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:33.871 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:33.871 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:34.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:34.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:34.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:34.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:34.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:34.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:34.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:34.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:34.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:34.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862790 2021-05-12T12:40:34.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862790 '!=' null 2021-05-12T12:40:34.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:34.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:34.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:34.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:34.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:34.959 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:34.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:34.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:34.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:34.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:34.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:35.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:35.321 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:35.321 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:35.321 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:35.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:35.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:35.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:35.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:35.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:35.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:35.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:35.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:35.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:36.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241868 2021-05-12T12:40:36.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241868 '!=' null 2021-05-12T12:40:36.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:36.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:36.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:36.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:36.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:36.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:36.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:36.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:36.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:36.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:36.402 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:36.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:36.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:36.757 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:36.757 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:37.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:37.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:37.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:37.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:37.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:37.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:37.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:37.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:37.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:37.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241832 2021-05-12T12:40:37.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241832 '!=' null 2021-05-12T12:40:37.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:37.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:37.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:37.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:37.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:37.830 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:37.830 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:37.830 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:37.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:37.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:37.831 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:38.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:38.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:38.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:38.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:38.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:38.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:38.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:38.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:38.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:38.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:38.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:38.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:38.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:39.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241832 2021-05-12T12:40:39.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241832 '!=' null 2021-05-12T12:40:39.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:39.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:39.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:39.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:39.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:39.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:39.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:39.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:39.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:39.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:39.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:39.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:39.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:39.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:39.762 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:40.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:40.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:40.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:40.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:40.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:40.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:40.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:40.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:40.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:40.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862761 2021-05-12T12:40:40.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862761 '!=' null 2021-05-12T12:40:40.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:40.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:40.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:40.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:40.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:40.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:40.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:40.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:40.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:40.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:40.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:41.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:41.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:41.302 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:41.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:41.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:41.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:41.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:41.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:41.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:41.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:41.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:41.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:41.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:42.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241885 2021-05-12T12:40:42.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241885 '!=' null 2021-05-12T12:40:42.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:42.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:42.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:42.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:42.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:42.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:42.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:42.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:42.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:42.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:42.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:42.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:42.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:42.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:42.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:43.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:43.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:43.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:43.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:43.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:43.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:43.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:43.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:43.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:43.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862826 2021-05-12T12:40:43.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862826 '!=' null 2021-05-12T12:40:43.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:43.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:43.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:43.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:43.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:43.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:43.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:43.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:43.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:43.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:43.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:44.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:44.263 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:44.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:44.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:44.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:44.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:44.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:44.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:44.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:44.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:44.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:44.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:44.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:45.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862826 2021-05-12T12:40:45.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862826 '!=' null 2021-05-12T12:40:45.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:45.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:45.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:45.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:45.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:45.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:45.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:45.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:45.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:45.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:45.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:45.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:45.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:45.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:45.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:46.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:46.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:46.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:46.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:46.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:46.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:46.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:46.241 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:46.241 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:46.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241940 2021-05-12T12:40:46.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241940 '!=' null 2021-05-12T12:40:46.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:46.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:46.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:46.825 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:46.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:46.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:46.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:46.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:46.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:46.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:46.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:47.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:47.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:47.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:47.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:47.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:47.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:47.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:47.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:47.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:47.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:47.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:47.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:47.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:48.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241930 2021-05-12T12:40:48.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241930 '!=' null 2021-05-12T12:40:48.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:48.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:48.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:48.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:48.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:48.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:48.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:48.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:48.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:48.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:48.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:48.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:48.660 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:48.661 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:48.661 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:49.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:49.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:49.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:49.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:49.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:49.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:49.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:49.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:49.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:49.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862964 2021-05-12T12:40:49.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862964 '!=' null 2021-05-12T12:40:49.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:49.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:49.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:49.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:49.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:49.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:49.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:49.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:49.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:49.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:49.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:50.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:50.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:50.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:50.256 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:50.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:50.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:50.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:50.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:50.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:50.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:50.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:50.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:50.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:51.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862964 2021-05-12T12:40:51.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862964 '!=' null 2021-05-12T12:40:51.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:51.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:51.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:51.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:51.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:51.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:51.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:51.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:51.338 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:51.339 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:51.339 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:51.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:51.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:51.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:51.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:52.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:52.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:52.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:52.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:52.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:52.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:52.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:52.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:52.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:52.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241957 2021-05-12T12:40:52.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241957 '!=' null 2021-05-12T12:40:52.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:52.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:52.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:52.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:52.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:52.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:52.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:52.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:52.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:52.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:52.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:53.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:53.129 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:53.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:53.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:53.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:53.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:53.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:53.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:53.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:53.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:53.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:53.668 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:53.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:54.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989264 2021-05-12T12:40:54.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989264 '!=' null 2021-05-12T12:40:54.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:54.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:54.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:54.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:54.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:54.256 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:54.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:54.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:54.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:54.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:54.258 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:54.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:54.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:54.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:54.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:55.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:55.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:55.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:55.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:55.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:55.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:55.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:55.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:55.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:55.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368325 2021-05-12T12:40:55.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368325 '!=' null 2021-05-12T12:40:55.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:55.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:55.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:55.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:55.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:55.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:55.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:55.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:55.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:55.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:55.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:56.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:56.088 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:56.089 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:56.089 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:56.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:56.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:56.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:56.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:56.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:56.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:56.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:56.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:56.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:57.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368325 2021-05-12T12:40:57.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368325 '!=' null 2021-05-12T12:40:57.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:57.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:57.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:57.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:57.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:57.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:57.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:57.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:57.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:57.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:57.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:57.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:57.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:57.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:57.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:58.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:58.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:58.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:58.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:58.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:58.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:58.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:58.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:58.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:58.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368334 2021-05-12T12:40:58.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368334 '!=' null 2021-05-12T12:40:58.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:58.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:58.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:58.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:58.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:58.717 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:58.717 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:58.717 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:58.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:58.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:58.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:59.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:59.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:59.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:59.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:59.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:59.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:59.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:59.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:59.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:59.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:59.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:59.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:59.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:00.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989317 2021-05-12T12:41:00.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989317 '!=' null 2021-05-12T12:41:00.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:00.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:00.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:00.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:00.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:00.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:00.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:00.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:00.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:00.192 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:00.192 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:00.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:00.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:00.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:00.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:01.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:01.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:01.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:01.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:01.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:01.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:01.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:01.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:01.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:01.516 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T12:41:01.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T12:41:01.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:01.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:01.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:01.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:01.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:01.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:01.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:01.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:01.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:01.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:01.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:01.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:01.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:01.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:01.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:02.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:02.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:02.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:02.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:02.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:02.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:02.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:02.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:02.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:02.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T12:41:02.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T12:41:02.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:02.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:02.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:03.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:03.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:03.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:03.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:03.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:03.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:03.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:03.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:03.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:03.451 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:03.451 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:03.451 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:03.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:03.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:03.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:03.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:03.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:03.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:03.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:03.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:03.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:04.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989313 2021-05-12T12:41:04.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989313 '!=' null 2021-05-12T12:41:04.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:04.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:04.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:04.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:04.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:04.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:04.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:04.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:04.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:04.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:04.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:04.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:04.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:04.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:04.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:05.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:05.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:05.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:05.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:05.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:05.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:05.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:05.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:05.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:05.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241985 2021-05-12T12:41:05.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241985 '!=' null 2021-05-12T12:41:05.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:05.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:05.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:06.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:06.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:06.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:06.021 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:06.022 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:06.022 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:06.022 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:06.022 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:06.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:06.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:06.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:06.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:06.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:06.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:06.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:06.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:06.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:06.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:06.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:06.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:06.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:07.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241991 2021-05-12T12:41:07.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241991 '!=' null 2021-05-12T12:41:07.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:07.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:07.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:07.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:07.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:07.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:07.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:07.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:07.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:07.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:07.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:07.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:07.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:07.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:07.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:08.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:08.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:08.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:08.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:08.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:08.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:08.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:08.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:08.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:08.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241991 2021-05-12T12:41:08.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241991 '!=' null 2021-05-12T12:41:08.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:08.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:08.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:08.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:08.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:08.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:08.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:08.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:08.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:08.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:08.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:09.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:09.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:09.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:09.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:09.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:09.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:09.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:09.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:09.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:09.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:09.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:09.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:09.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:10.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989343 2021-05-12T12:41:10.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989343 '!=' null 2021-05-12T12:41:10.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:10.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:10.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:10.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:10.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:10.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:10.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:10.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:10.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:10.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:10.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:10.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:10.738 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:10.738 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:10.738 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:11.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:11.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:11.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:11.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:11.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:11.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:11.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:11.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:11.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:11.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368348 2021-05-12T12:41:11.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368348 '!=' null 2021-05-12T12:41:11.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:11.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:11.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:11.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:11.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:11.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:11.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:11.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:11.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:11.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:11.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:12.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:12.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:12.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:12.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:12.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:12.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:12.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:12.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:12.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:12.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:12.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:12.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:12.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:13.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368348 2021-05-12T12:41:13.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368348 '!=' null 2021-05-12T12:41:13.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:13.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:13.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:13.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:13.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:13.263 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:13.263 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:13.263 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:13.263 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:13.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:13.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:13.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:13.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:13.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:13.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:14.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:14.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:14.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:14.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:14.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:14.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:14.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:14.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:14.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:14.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989376 2021-05-12T12:41:14.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989376 '!=' null 2021-05-12T12:41:14.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:14.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:14.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:14.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:14.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:14.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:14.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:14.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:14.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:14.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:14.702 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:15.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:15.067 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:15.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:15.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:15.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:15.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:15.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:15.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:15.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:15.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:15.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:15.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:15.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:16.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368370 2021-05-12T12:41:16.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368370 '!=' null 2021-05-12T12:41:16.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:16.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:16.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:16.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:16.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:16.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:16.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:16.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:16.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:16.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:16.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:16.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:16.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:16.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:16.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:17.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:17.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:17.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:17.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:17.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:17.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:17.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:17.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:17.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:17.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368360 2021-05-12T12:41:17.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368360 '!=' null 2021-05-12T12:41:17.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:17.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:17.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:17.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:17.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:17.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:17.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:17.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:17.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:17.606 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:17.606 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:17.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:17.966 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:17.966 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:17.966 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:18.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:18.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:18.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:18.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:18.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:18.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:18.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:18.457 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:18.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:18.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368360 2021-05-12T12:41:18.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368360 '!=' null 2021-05-12T12:41:18.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:18.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:18.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:19.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:19.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:19.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:19.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:19.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:19.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:19.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:19.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:19.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:19.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:19.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:19.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:19.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:19.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:19.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:19.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:19.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:19.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:19.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:19.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:19.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:20.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863031 2021-05-12T12:41:20.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863031 '!=' null 2021-05-12T12:41:20.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:20.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:20.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:20.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:20.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:20.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:20.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:20.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:20.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:20.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:20.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:20.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:20.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:20.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:20.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:21.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:21.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:21.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:21.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:21.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:21.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:21.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:21.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:21.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:21.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115694 2021-05-12T12:41:21.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115694 '!=' null 2021-05-12T12:41:21.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:21.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:21.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:22.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:22.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:22.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:22.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:22.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:22.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:22.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:22.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:22.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:22.413 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:22.413 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:22.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:22.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:22.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:22.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:22.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:22.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:22.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:22.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:22.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:22.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:23.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736719 2021-05-12T12:41:23.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736719 '!=' null 2021-05-12T12:41:23.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:23.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:23.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:23.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:23.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:23.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:23.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:23.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:23.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:23.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:23.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:23.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:23.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:23.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:23.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:24.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:24.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:24.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:24.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:24.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:24.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:24.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:24.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:24.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:24.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736719 2021-05-12T12:41:24.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736719 '!=' null 2021-05-12T12:41:24.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:24.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:24.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:24.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:24.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:24.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:24.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:24.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:24.973 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:24.973 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:24.973 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:25.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:25.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:25.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:25.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:25.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:25.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:25.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:25.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:25.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:25.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:25.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:25.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:25.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:26.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115691 2021-05-12T12:41:26.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115691 '!=' null 2021-05-12T12:41:26.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:26.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:26.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:26.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:26.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:26.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:26.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:26.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:26.559 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:26.559 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:26.559 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:26.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:26.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:26.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:26.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:27.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:27.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:27.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:27.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:27.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:27.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:27.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:27.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:27.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:27.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115694 2021-05-12T12:41:27.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115694 '!=' null 2021-05-12T12:41:27.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:27.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:27.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:28.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:28.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:28.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:28.041 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:28.041 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:28.041 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:28.041 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:28.042 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:28.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:28.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:28.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:28.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:28.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:28.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:28.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:28.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:28.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:28.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:28.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:28.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:28.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:29.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863062 2021-05-12T12:41:29.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863062 '!=' null 2021-05-12T12:41:29.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:29.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:29.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:29.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:29.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:29.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:29.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:29.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:29.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:29.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:29.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:29.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:29.901 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:29.901 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:29.901 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:30.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:30.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:30.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:30.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:30.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:30.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:30.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:30.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:30.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:30.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863062 2021-05-12T12:41:30.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863062 '!=' null 2021-05-12T12:41:30.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:30.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:30.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:30.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:30.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:30.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:30.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:30.983 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:30.983 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:30.983 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:30.983 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:31.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:31.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:31.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:31.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:31.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:31.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:31.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:31.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:31.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:31.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:31.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:31.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:31.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:32.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368370 2021-05-12T12:41:32.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368370 '!=' null 2021-05-12T12:41:32.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:32.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:32.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:32.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:32.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:32.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:32.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:32.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:32.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:32.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:32.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:32.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:32.863 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:32.863 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:32.863 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:33.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:33.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:33.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:33.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:33.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:33.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:33.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:33.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:33.356 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:33.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989384 2021-05-12T12:41:33.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989384 '!=' null 2021-05-12T12:41:33.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:33.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:33.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:34.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:34.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:34.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:34.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:34.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:34.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:34.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:34.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:34.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:34.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:34.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:34.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:35.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:35.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:35.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:35.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:35.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:35.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:35.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:35.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:35.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:35.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368333 2021-05-12T12:41:35.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368333 '!=' null 2021-05-12T12:41:35.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:35.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:35.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:35.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:35.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:35.598 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:35.598 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:35.598 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:35.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:35.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:35.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:35.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:35.991 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:35.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:35.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:36.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:36.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:36.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:36.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:36.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:36.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:36.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:36.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:36.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:36.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368333 2021-05-12T12:41:36.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368333 '!=' null 2021-05-12T12:41:36.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:36.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:36.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:37.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:37.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:37.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:37.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:37.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:37.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:37.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:37.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:37.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:37.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:37.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:37.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:37.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:37.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:37.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:37.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:37.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:37.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:37.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:37.966 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:37.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:38.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368324 2021-05-12T12:41:38.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368324 '!=' null 2021-05-12T12:41:38.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:38.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:38.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:38.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:38.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:38.557 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:38.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:38.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:38.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:38.559 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:38.559 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:38.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:38.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:38.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:38.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:39.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:39.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:39.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:39.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:39.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:39.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:39.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:39.407 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:39.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:39.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115594 2021-05-12T12:41:39.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115594 '!=' null 2021-05-12T12:41:39.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:39.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:39.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:39.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:39.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:40.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:40.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:40.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:40.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:40.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:40.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:40.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:40.354 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:40.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:40.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:40.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:40.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:40.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:40.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:40.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:40.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:40.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:40.850 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:40.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:41.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368282 2021-05-12T12:41:41.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368282 '!=' null 2021-05-12T12:41:41.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:41.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:41.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:41.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:41.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:41.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:41.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:41.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:41.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:41.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:41.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:41.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:41.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:41.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:41.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:42.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:42.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:42.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:42.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:42.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:42.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:42.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:42.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:42.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:42.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368282 2021-05-12T12:41:42.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368282 '!=' null 2021-05-12T12:41:42.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:42.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:42.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:42.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:42.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:42.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:42.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:42.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:42.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:42.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:42.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:43.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:43.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:43.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:43.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:43.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:43.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:43.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:43.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:43.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:43.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:43.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:43.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:43.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:44.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989206 2021-05-12T12:41:44.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989206 '!=' null 2021-05-12T12:41:44.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:44.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:44.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:44.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:44.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:44.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:44.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:44.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:44.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:44.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:44.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:44.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:44.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:44.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:44.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:45.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:45.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:45.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:45.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:45.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:45.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:45.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:45.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:45.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:45.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368239 2021-05-12T12:41:45.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368239 '!=' null 2021-05-12T12:41:45.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:45.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:45.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:45.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:45.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:45.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:45.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:45.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:45.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:45.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:45.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:46.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:46.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:46.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:46.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:46.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:46.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:46.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:46.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:46.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:46.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:46.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:46.675 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:46.676 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:47.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368239 2021-05-12T12:41:47.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368239 '!=' null 2021-05-12T12:41:47.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:47.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:47.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:47.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:47.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:47.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:47.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:47.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:47.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:47.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:47.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:47.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:47.642 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:47.642 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:47.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:48.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:48.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:48.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:48.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:48.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:48.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:48.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:48.140 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:48.140 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:48.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368269 2021-05-12T12:41:48.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368269 '!=' null 2021-05-12T12:41:48.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:48.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:48.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:48.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:48.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:48.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:48.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:48.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:48.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:48.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:48.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:49.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:49.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:49.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:49.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:49.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:49.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:49.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:49.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:49.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:49.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:49.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:49.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:49.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:50.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989257 2021-05-12T12:41:50.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989257 '!=' null 2021-05-12T12:41:50.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:50.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:50.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:50.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:50.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:50.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:50.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:50.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:50.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:50.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:50.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:50.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:50.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:50.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:50.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:51.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:51.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:51.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:51.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:51.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:51.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:51.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:51.173 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:51.173 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:51.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241983 2021-05-12T12:41:51.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241983 '!=' null 2021-05-12T12:41:51.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:51.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:51.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:51.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:51.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:51.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:51.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:51.839 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:51.839 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:51.839 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:51.840 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:52.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:52.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:52.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:52.218 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:52.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:52.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:52.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:52.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:52.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:52.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:52.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:52.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:52.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:53.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241983 2021-05-12T12:41:53.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241983 '!=' null 2021-05-12T12:41:53.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:53.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:53.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:53.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:53.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:53.281 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:53.281 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:53.282 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:53.282 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:53.282 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:53.282 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:53.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:53.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:53.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:53.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:54.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:54.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:54.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:54.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:54.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:54.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:54.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:54.129 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:54.129 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:54.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862999 2021-05-12T12:41:54.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862999 '!=' null 2021-05-12T12:41:54.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:54.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:54.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:54.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:54.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:54.704 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:54.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:54.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:54.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:54.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:54.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:55.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:55.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:55.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:55.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:55.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:55.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:55.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:55.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:55.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:55.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:55.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:55.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:55.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:56.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242034 2021-05-12T12:41:56.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242034 '!=' null 2021-05-12T12:41:56.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:56.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:56.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:56.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:56.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:56.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:56.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:56.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:56.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:56.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:56.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:56.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:56.541 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:56.541 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:56.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:57.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:57.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:57.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:57.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:57.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:57.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:57.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:57.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:57.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:57.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242027 2021-05-12T12:41:57.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242027 '!=' null 2021-05-12T12:41:57.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:57.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:57.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:57.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:57.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:57.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:57.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:57.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:57.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:57.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:57.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:57.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:57.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:57.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:57.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:58.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:58.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:58.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:58.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:58.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:58.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:58.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:58.474 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:58.474 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:58.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242027 2021-05-12T12:41:58.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242027 '!=' null 2021-05-12T12:41:58.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:58.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:58.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:59.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:59.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:59.051 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:59.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:59.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:59.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:59.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:59.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:59.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:59.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:59.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:59.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:59.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:59.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:59.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:59.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:59.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:59.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:59.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:59.901 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:59.901 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:00.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736662 2021-05-12T12:42:00.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736662 '!=' null 2021-05-12T12:42:00.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:00.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:00.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:00.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:00.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:00.611 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:00.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:00.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:00.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:00.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:00.613 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:00.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:00.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:00.976 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:00.976 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:01.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:01.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:01.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:01.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:01.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:01.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:01.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:01.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:01.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:01.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115646 2021-05-12T12:42:01.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115646 '!=' null 2021-05-12T12:42:01.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:01.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:01.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:02.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:02.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:02.062 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:02.063 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:02.063 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:02.063 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:02.063 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:02.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:02.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:02.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:02.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:02.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:02.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:02.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:02.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:02.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:02.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:02.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:02.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:02.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:02.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:03.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862986 2021-05-12T12:42:03.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862986 '!=' null 2021-05-12T12:42:03.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:03.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:03.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:03.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:03.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:03.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:03.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:03.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:03.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:03.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:03.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:03.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:03.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:03.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:03.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:04.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:04.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:04.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:04.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:04.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:04.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:04.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:04.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:04.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:04.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862986 2021-05-12T12:42:04.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862986 '!=' null 2021-05-12T12:42:04.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:04.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:04.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:05.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:05.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:05.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:05.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:05.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:05.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:05.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:05.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:05.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:05.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:05.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:05.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:05.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:05.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:05.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:05.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:05.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:05.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:05.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:05.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:05.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:06.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241976 2021-05-12T12:42:06.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241976 '!=' null 2021-05-12T12:42:06.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:06.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:06.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:06.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:06.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:06.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:06.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:06.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:06.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:06.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:06.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:06.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:06.813 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:06.813 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:06.813 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:07.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:07.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:07.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:07.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:07.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:07.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:07.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:07.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:07.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:07.829 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241988 2021-05-12T12:42:07.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241988 '!=' null 2021-05-12T12:42:07.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:07.830 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:07.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:07.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:07.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:07.932 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:07.932 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:07.932 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:07.932 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:07.933 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:07.933 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:08.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:08.289 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:08.289 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:08.289 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:08.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:08.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:08.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:08.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:08.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:08.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:08.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:08.778 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:08.778 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:09.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863002 2021-05-12T12:42:09.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863002 '!=' null 2021-05-12T12:42:09.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:09.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:09.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:09.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:09.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:09.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:09.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:09.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:09.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:09.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:09.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:09.744 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:09.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:09.746 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:09.746 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:10.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:10.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:10.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:10.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:10.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:10.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:10.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:10.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:10.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:10.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863002 2021-05-12T12:42:10.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863002 '!=' null 2021-05-12T12:42:10.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:10.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:10.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:10.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:10.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:10.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:10.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:10.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:10.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:10.849 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:10.849 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:11.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:11.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:11.203 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:11.204 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:11.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:11.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:11.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:11.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:11.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:11.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:11.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:11.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:11.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:12.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368354 2021-05-12T12:42:12.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368354 '!=' null 2021-05-12T12:42:12.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:12.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:12.176 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:12.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:12.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:12.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:12.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:12.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:12.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:12.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:12.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:12.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:12.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:12.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:12.644 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:13.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:13.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:13.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:13.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:13.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:13.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:13.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:13.138 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:13.138 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:13.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115674 2021-05-12T12:42:13.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115674 '!=' null 2021-05-12T12:42:13.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:13.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:13.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:13.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:13.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:13.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:13.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:13.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:13.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:13.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:13.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:14.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:14.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:14.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:14.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:14.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:14.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:14.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:14.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:14.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:14.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:14.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:14.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:14.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:15.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115674 2021-05-12T12:42:15.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115674 '!=' null 2021-05-12T12:42:15.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:15.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:15.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:15.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:15.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:15.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:15.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:15.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:15.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:15.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:15.232 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:15.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:15.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:15.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:15.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:16.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:16.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:16.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:16.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:16.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:16.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:16.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:16.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:16.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:16.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T12:42:16.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T12:42:16.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:16.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:16.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:16.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:16.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:16.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:16.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:16.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:16.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:16.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:16.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:17.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:17.026 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:17.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:17.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:17.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:17.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:17.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:17.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:17.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:17.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:17.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:17.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:17.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:18.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368322 2021-05-12T12:42:18.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368322 '!=' null 2021-05-12T12:42:18.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:18.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:18.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:18.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:18.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:18.133 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:18.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:18.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:18.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:18.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:18.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:18.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:18.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:18.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:18.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:19.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:19.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:19.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:19.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:19.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:19.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:19.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:19.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:19.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:19.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989324 2021-05-12T12:42:19.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989324 '!=' null 2021-05-12T12:42:19.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:19.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:19.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:19.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:19.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:19.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:19.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:19.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:19.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:19.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:19.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:19.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:19.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:19.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:19.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:20.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:20.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:20.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:20.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:20.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:20.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:20.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:20.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:20.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:20.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989324 2021-05-12T12:42:20.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989324 '!=' null 2021-05-12T12:42:20.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:20.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:20.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:21.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:21.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:21.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:21.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:21.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:21.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:21.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:21.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:21.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:21.436 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:21.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:21.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:21.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:21.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:21.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:21.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:21.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:21.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:21.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:21.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:21.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:22.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368296 2021-05-12T12:42:22.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368296 '!=' null 2021-05-12T12:42:22.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:22.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:22.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:22.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:22.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:22.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:22.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:22.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:22.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:22.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:22.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:22.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:22.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:22.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:22.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:23.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:23.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:23.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:23.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:23.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:23.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:23.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:23.430 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:23.430 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:23.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989250 2021-05-12T12:42:23.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989250 '!=' null 2021-05-12T12:42:23.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:23.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:23.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:24.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:24.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:24.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:24.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:24.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:24.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:24.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:24.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:24.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:24.387 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:24.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:24.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:24.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:24.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:24.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:24.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:24.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:24.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:24.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:24.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:24.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:25.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241946 2021-05-12T12:42:25.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241946 '!=' null 2021-05-12T12:42:25.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:25.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:25.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:25.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:25.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:25.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:25.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:25.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:25.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:25.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:25.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:25.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:25.832 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:25.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:25.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:26.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:26.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:26.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:26.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:26.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:26.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:26.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:26.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:26.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:26.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241946 2021-05-12T12:42:26.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241946 '!=' null 2021-05-12T12:42:26.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:26.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:26.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:26.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:26.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:26.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:26.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:26.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:26.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:26.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:26.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:27.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:27.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:27.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:27.352 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:27.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:27.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:27.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:27.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:27.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:27.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:27.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:27.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:27.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:28.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241978 2021-05-12T12:42:28.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241978 '!=' null 2021-05-12T12:42:28.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:28.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:28.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:28.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:28.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:28.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:28.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:28.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:28.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:28.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:28.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:28.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:28.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:28.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:28.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:29.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:29.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:29.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:29.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:29.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:29.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:29.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:29.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:29.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:29.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862899 2021-05-12T12:42:29.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862899 '!=' null 2021-05-12T12:42:29.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:29.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:29.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:29.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:29.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:29.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:29.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:29.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:29.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:29.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:29.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:30.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:30.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:30.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:30.281 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:30.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:30.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:30.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:30.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:30.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:30.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:30.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:30.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:30.777 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:31.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241869 2021-05-12T12:42:31.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241869 '!=' null 2021-05-12T12:42:31.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:31.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:31.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:31.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:31.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:31.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:31.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:31.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:31.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:31.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:31.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:31.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:31.763 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:31.763 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:31.763 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:32.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:32.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:32.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:32.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:32.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:32.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:32.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:32.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:32.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:32.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241869 2021-05-12T12:42:32.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241869 '!=' null 2021-05-12T12:42:32.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:32.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:32.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:32.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:32.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:32.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:32.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:32.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:32.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:32.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:32.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:33.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:33.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:33.210 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:33.211 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:33.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:33.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:33.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:33.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:33.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:33.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:33.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:33.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:33.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:34.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862874 2021-05-12T12:42:34.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862874 '!=' null 2021-05-12T12:42:34.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:34.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:34.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:34.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:34.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:34.290 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:34.290 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:34.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:34.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:34.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:34.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:34.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:34.644 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:34.644 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:34.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:35.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:35.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:35.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:35.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:35.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:35.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:35.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:35.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:35.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:35.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241923 2021-05-12T12:42:35.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241923 '!=' null 2021-05-12T12:42:35.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:35.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:35.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:35.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:35.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:35.743 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:35.744 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:35.744 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:35.744 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:35.744 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:35.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:36.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:36.100 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:36.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:36.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:36.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:36.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:36.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:36.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:36.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:36.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:36.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:36.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:36.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:37.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241923 2021-05-12T12:42:37.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241923 '!=' null 2021-05-12T12:42:37.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:37.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:37.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:37.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:37.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:37.196 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:37.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:37.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:37.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:37.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:37.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:37.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:37.553 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:37.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:37.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:38.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:38.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:38.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:38.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:38.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:38.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:38.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:38.063 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:38.063 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:38.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241952 2021-05-12T12:42:38.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241952 '!=' null 2021-05-12T12:42:38.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:38.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:38.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:38.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:38.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:38.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:38.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:38.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:38.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:38.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:38.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:39.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:39.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:39.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:39.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:39.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:39.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:39.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:39.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:39.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:39.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:39.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:39.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:39.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:40.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862917 2021-05-12T12:42:40.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862917 '!=' null 2021-05-12T12:42:40.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:40.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:40.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:40.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:40.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:40.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:40.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:40.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:40.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:40.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:40.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:40.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:40.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:40.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:40.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:41.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:41.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:41.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:41.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:41.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:41.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:41.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:41.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:41.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:41.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241983 2021-05-12T12:42:41.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241983 '!=' null 2021-05-12T12:42:41.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:41.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:41.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:41.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:41.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:41.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:41.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:41.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:41.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:41.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:41.637 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:42.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:42.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:42.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:42.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:42.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:42.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:42.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:42.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:42.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:42.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:42.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:42.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:42.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:43.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241983 2021-05-12T12:42:43.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241983 '!=' null 2021-05-12T12:42:43.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:43.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:43.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:43.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:43.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:43.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:43.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:43.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:43.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:43.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:43.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:43.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:43.549 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:43.549 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:43.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:44.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:44.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:44.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:44.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:44.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:44.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:44.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:44.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:44.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:44.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863058 2021-05-12T12:42:44.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863058 '!=' null 2021-05-12T12:42:44.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:44.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:44.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:44.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:44.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:44.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:44.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:44.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:44.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:44.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:44.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:45.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:45.156 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:45.156 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:45.156 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:45.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:45.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:45.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:45.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:45.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:45.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:45.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:45.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:45.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:46.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242036 2021-05-12T12:42:46.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242036 '!=' null 2021-05-12T12:42:46.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:46.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:46.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:46.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:46.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:46.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:46.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:46.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:46.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:46.321 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:46.321 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:46.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:46.676 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:46.676 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:46.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:47.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:47.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:47.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:47.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:47.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:47.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:47.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:47.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:47.168 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:47.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242051 2021-05-12T12:42:47.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242051 '!=' null 2021-05-12T12:42:47.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:47.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:47.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:47.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:47.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:47.763 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:47.763 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:47.763 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:47.763 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:47.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:47.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:48.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:48.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:48.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:48.119 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:48.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:48.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:48.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:48.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:48.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:48.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:48.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:48.608 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:48.609 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:49.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242051 2021-05-12T12:42:49.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242051 '!=' null 2021-05-12T12:42:49.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:49.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:49.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:49.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:49.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:49.206 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:49.206 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:49.206 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:49.207 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:49.207 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:49.208 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:49.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:49.560 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:49.560 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:49.560 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:50.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:50.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:50.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:50.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:50.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:50.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:50.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:50.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:50.052 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:50.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989351 2021-05-12T12:42:50.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989351 '!=' null 2021-05-12T12:42:50.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:50.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:50.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:50.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:50.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:50.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:50.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:50.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:50.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:50.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:50.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:51.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:51.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:51.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:51.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:51.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:51.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:51.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:51.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:51.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:51.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:51.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:51.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:51.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:52.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368304 2021-05-12T12:42:52.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368304 '!=' null 2021-05-12T12:42:52.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:52.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:52.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:52.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:52.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:52.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:52.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:52.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:52.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:52.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:52.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:52.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:52.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:52.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:52.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:52.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:52.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:52.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:52.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:52.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:52.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:52.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:52.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:52.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:53.516 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989245 2021-05-12T12:42:53.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989245 '!=' null 2021-05-12T12:42:53.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:53.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:53.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:53.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:53.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:53.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:53.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:53.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:53.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:53.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:53.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:53.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:53.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:53.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:53.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:54.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:54.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:54.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:54.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:54.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:54.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:54.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:54.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:54.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:54.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989245 2021-05-12T12:42:54.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989245 '!=' null 2021-05-12T12:42:54.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:54.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:54.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:55.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:55.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:55.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:55.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:55.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:55.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:55.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:55.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:55.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:55.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:55.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:55.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:55.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:55.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:55.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:55.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:55.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:55.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:55.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:55.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:55.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:56.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368286 2021-05-12T12:42:56.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368286 '!=' null 2021-05-12T12:42:56.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:56.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:56.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:56.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:56.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:56.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:56.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:56.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:56.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:56.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:56.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:56.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:56.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:56.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:56.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:57.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:57.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:57.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:57.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:57.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:57.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:57.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:57.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:57.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:57.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368286 2021-05-12T12:42:57.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368286 '!=' null 2021-05-12T12:42:57.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:57.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:57.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:57.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:57.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:57.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:57.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:57.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:57.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:57.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:57.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:58.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:58.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:58.327 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:58.327 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:58.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:58.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:58.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:58.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:58.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:58.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:58.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:58.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:58.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:59.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989199 2021-05-12T12:42:59.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989199 '!=' null 2021-05-12T12:42:59.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:59.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:59.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:59.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:59.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:59.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:59.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:59.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:59.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:59.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:59.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:59.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:59.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:59.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:59.825 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:00.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:00.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:00.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:00.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:00.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:00.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:00.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:00.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:00.311 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:00.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989199 2021-05-12T12:43:00.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989199 '!=' null 2021-05-12T12:43:00.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:00.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:00.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:00.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:00.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:00.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:00.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:00.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:00.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:00.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:00.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:01.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:01.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:01.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:01.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:01.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:01.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:01.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:01.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:01.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:01.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:01.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:01.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:01.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:02.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241929 2021-05-12T12:43:02.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241929 '!=' null 2021-05-12T12:43:02.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:02.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:02.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:02.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:02.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:02.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:02.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:02.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:02.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:02.332 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:02.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:02.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:02.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:02.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:02.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:03.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:03.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:03.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:03.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:03.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:03.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:03.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:03.190 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:03.190 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:03.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989258 2021-05-12T12:43:03.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989258 '!=' null 2021-05-12T12:43:03.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:03.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:03.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:03.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:03.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:03.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:03.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:03.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:03.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:03.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:03.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:04.142 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:04.143 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:04.143 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:04.144 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:04.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:04.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:04.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:04.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:04.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:04.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:04.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:04.626 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:04.626 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:05.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989258 2021-05-12T12:43:05.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989258 '!=' null 2021-05-12T12:43:05.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:05.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:05.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:05.225 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:05.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:05.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:05.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:05.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:05.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:05.228 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:05.228 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:05.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:05.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:05.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:05.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:06.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:06.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:06.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:06.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:06.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:06.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:06.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:06.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:06.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:06.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368320 2021-05-12T12:43:06.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368320 '!=' null 2021-05-12T12:43:06.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:06.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:06.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:06.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:06.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:06.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:06.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:06.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:06.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:06.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:06.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:07.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:07.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:07.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:07.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:07.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:07.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:07.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:07.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:07.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:07.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:07.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:07.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:07.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:08.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368302 2021-05-12T12:43:08.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368302 '!=' null 2021-05-12T12:43:08.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:08.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:08.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:08.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:08.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:08.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:08.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:08.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:08.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:08.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:08.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:08.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:08.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:08.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:08.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:09.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:09.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:09.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:09.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:09.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:09.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:09.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:09.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:09.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:09.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989284 2021-05-12T12:43:09.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989284 '!=' null 2021-05-12T12:43:09.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:09.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:09.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:09.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:09.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:09.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:09.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:09.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:09.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:09.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:09.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:10.016 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:10.017 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:10.017 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:10.017 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:10.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:10.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:10.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:10.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:10.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:10.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:10.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:10.507 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:10.507 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:11.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989284 2021-05-12T12:43:11.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989284 '!=' null 2021-05-12T12:43:11.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:11.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:11.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:11.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:11.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:11.229 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:11.229 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:11.230 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:11.230 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:11.230 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:11.230 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:11.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:11.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:11.583 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:11.583 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:12.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:12.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:12.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:12.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:12.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:12.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:12.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:12.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:12.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:12.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368325 2021-05-12T12:43:12.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368325 '!=' null 2021-05-12T12:43:12.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:12.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:12.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:12.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:12.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:12.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:12.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:12.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:12.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:12.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:12.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:13.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:13.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:13.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:13.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:13.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:13.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:13.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:13.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:13.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:13.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:13.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:13.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:13.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:14.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989333 2021-05-12T12:43:14.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989333 '!=' null 2021-05-12T12:43:14.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:14.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:14.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:14.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:14.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:14.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:14.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:14.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:14.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:14.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:14.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:14.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:14.475 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:14.475 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:14.475 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:14.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:14.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:14.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:14.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:14.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:14.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:14.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:14.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:14.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:15.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241995 2021-05-12T12:43:15.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241995 '!=' null 2021-05-12T12:43:15.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:15.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:15.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:15.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:15.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:15.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:15.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:15.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:15.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:15.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:15.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:16.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:16.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:16.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:16.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:16.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:16.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:16.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:16.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:16.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:16.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:16.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:16.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:16.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:17.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241995 2021-05-12T12:43:17.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241995 '!=' null 2021-05-12T12:43:17.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:17.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:17.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:17.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:17.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:17.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:17.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:17.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:17.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:17.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:17.254 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:17.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:17.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:17.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:17.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:18.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:18.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:18.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:18.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:18.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:18.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:18.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:18.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:18.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:18.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241930 2021-05-12T12:43:18.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241930 '!=' null 2021-05-12T12:43:18.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:18.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:18.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:18.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:18.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:18.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:18.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:18.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:18.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:18.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:18.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:19.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:19.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:19.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:19.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:19.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:19.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:19.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:19.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:19.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:19.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:19.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:19.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:19.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:20.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736581 2021-05-12T12:43:20.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736581 '!=' null 2021-05-12T12:43:20.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:20.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:20.030 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:20.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:20.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:20.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:20.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:20.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:20.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:20.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:20.133 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:20.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:20.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:20.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:20.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:20.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:20.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:20.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:20.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:21.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:21.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:21.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115565 2021-05-12T12:43:21.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115565 '!=' null 2021-05-12T12:43:21.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:21.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:21.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:21.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:21.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:21.601 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:21.601 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:21.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:21.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:21.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:21.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:21.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:21.955 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:21.956 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:21.956 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:22.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:22.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:22.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:22.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:22.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:22.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:22.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:22.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:22.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:22.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115565 2021-05-12T12:43:22.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115565 '!=' null 2021-05-12T12:43:22.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:22.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:22.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:23.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:23.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:23.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:23.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:23.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:23.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:23.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:23.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:23.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:23.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:23.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:23.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:23.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:23.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:23.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:23.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:23.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:23.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:23.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:23.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:23.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:24.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862851 2021-05-12T12:43:24.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862851 '!=' null 2021-05-12T12:43:24.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:24.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:24.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:24.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:24.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:24.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:24.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:24.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:24.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:24.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:24.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:24.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:24.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:24.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:24.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:25.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:25.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:25.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:25.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:25.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:25.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:25.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:25.335 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:25.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:25.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241886 2021-05-12T12:43:25.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241886 '!=' null 2021-05-12T12:43:25.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:25.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:25.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:25.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:25.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:25.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:25.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:25.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:25.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:25.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:25.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:26.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:26.308 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:26.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:26.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:26.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:26.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:26.793 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:26.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:26.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:26.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:26.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:26.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:26.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:27.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241886 2021-05-12T12:43:27.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241886 '!=' null 2021-05-12T12:43:27.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:27.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:27.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:27.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:27.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:27.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:27.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:27.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:27.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:27.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:27.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:27.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:27.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:27.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:27.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:28.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:28.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:28.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:28.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:28.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:28.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:28.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:28.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:28.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:28.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241924 2021-05-12T12:43:28.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241924 '!=' null 2021-05-12T12:43:28.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:28.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:28.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:28.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:28.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:28.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:28.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:28.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:28.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:28.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:28.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:29.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:29.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:29.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:29.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:29.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:29.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:29.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:29.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:29.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:29.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:29.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:29.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:29.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:30.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862970 2021-05-12T12:43:30.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862970 '!=' null 2021-05-12T12:43:30.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:30.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:30.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:30.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:30.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:30.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:30.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:30.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:30.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:30.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:30.300 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:30.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:30.676 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:30.676 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:30.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:31.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:31.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:31.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:31.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:31.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:31.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:31.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:31.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:31.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:31.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368306 2021-05-12T12:43:31.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368306 '!=' null 2021-05-12T12:43:31.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:31.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:31.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:31.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:31.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:31.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:31.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:31.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:31.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:31.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:31.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:32.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:32.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:32.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:32.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:32.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:32.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:32.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:32.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:32.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:32.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:32.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:32.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:32.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:33.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368306 2021-05-12T12:43:33.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368306 '!=' null 2021-05-12T12:43:33.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:33.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:33.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:33.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:33.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:33.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:33.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:33.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:33.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:33.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:33.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:33.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:33.611 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:33.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:33.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:34.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:34.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:34.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:34.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:34.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:34.097 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:34.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:34.098 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:34.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:34.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115664 2021-05-12T12:43:34.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115664 '!=' null 2021-05-12T12:43:34.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:34.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:34.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:34.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:34.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:34.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:34.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:34.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:34.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:34.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:34.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:35.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:35.050 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:35.050 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:35.051 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:35.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:35.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:35.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:35.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:35.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:35.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:35.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:35.548 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:35.548 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:36.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368355 2021-05-12T12:43:36.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368355 '!=' null 2021-05-12T12:43:36.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:36.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:36.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:36.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:36.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:36.123 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:36.123 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:36.123 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:36.124 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:36.124 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:36.124 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:36.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:36.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:36.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:36.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:36.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:36.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:36.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:36.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:36.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:36.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:36.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:36.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:36.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:37.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368377 2021-05-12T12:43:37.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368377 '!=' null 2021-05-12T12:43:37.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:37.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:37.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:37.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:37.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:37.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:37.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:37.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:37.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:37.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:37.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:37.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:37.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:37.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:37.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:38.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:38.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:38.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:38.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:38.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:38.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:38.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:38.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:38.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:38.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368377 2021-05-12T12:43:38.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368377 '!=' null 2021-05-12T12:43:38.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:38.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:38.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:39.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:39.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:39.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:39.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:39.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:39.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:39.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:39.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:39.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:39.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:39.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:39.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:39.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:39.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:39.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:39.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:39.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:39.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:39.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:39.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:39.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:40.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989359 2021-05-12T12:43:40.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989359 '!=' null 2021-05-12T12:43:40.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:40.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:40.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:40.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:40.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:40.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:40.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:40.581 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:40.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:40.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:40.582 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:40.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:40.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:40.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:40.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:41.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:41.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:41.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:41.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:41.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:41.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:41.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:41.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:41.420 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:41.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368313 2021-05-12T12:43:41.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368313 '!=' null 2021-05-12T12:43:41.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:41.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:41.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:42.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:42.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:42.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:42.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:42.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:42.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:42.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:42.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:42.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:42.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:42.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:42.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:42.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:42.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:42.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:42.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:42.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:42.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:42.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:42.863 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:42.864 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:43.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368313 2021-05-12T12:43:43.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368313 '!=' null 2021-05-12T12:43:43.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:43.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:43.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:43.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:43.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:43.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:43.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:43.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:43.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:43.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:43.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:43.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:43.808 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:43.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:43.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:44.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:44.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:44.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:44.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:44.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:44.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:44.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:44.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:44.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:44.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989247 2021-05-12T12:43:44.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989247 '!=' null 2021-05-12T12:43:44.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:44.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:44.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:44.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:44.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:44.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:44.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:44.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:44.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:44.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:44.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:45.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:45.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:45.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:45.256 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:46.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:46.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:46.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:46.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:46.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:46.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:46.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:46.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:46.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:46.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241929 2021-05-12T12:43:46.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241929 '!=' null 2021-05-12T12:43:46.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:46.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:46.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:46.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:46.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:46.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:46.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:46.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:46.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:46.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:46.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:46.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:46.943 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:46.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:46.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:47.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:47.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:47.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:47.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:47.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:47.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:47.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:47.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:47.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:47.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241881 2021-05-12T12:43:47.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241881 '!=' null 2021-05-12T12:43:47.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:47.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:47.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:48.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:48.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:48.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:48.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:48.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:48.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:48.067 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:48.067 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:48.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:48.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:48.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:48.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:48.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:48.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:48.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:48.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:48.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:49.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862786 2021-05-12T12:43:49.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862786 '!=' null 2021-05-12T12:43:49.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:49.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:49.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:49.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:49.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:49.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:49.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:49.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:49.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:49.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:49.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:49.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:50.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:50.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:50.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:50.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:50.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:50.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:50.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:50.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:50.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:50.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:50.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:50.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:50.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862786 2021-05-12T12:43:50.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862786 '!=' null 2021-05-12T12:43:50.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:50.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:50.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:51.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:51.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:51.029 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:51.029 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:51.029 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:51.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:51.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:51.030 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:51.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:51.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:51.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:51.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:51.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:51.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:51.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:51.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:51.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:51.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:51.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:51.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:51.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:52.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241893 2021-05-12T12:43:52.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241893 '!=' null 2021-05-12T12:43:52.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:52.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:52.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:52.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:52.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:52.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:52.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:52.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:52.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:52.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:52.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:52.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:52.883 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:52.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:52.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:53.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:53.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:53.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:53.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:53.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:53.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:53.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:53.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:53.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:53.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862855 2021-05-12T12:43:53.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862855 '!=' null 2021-05-12T12:43:53.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:53.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:53.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:54.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:54.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:54.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:54.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:54.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:54.061 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:54.061 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:54.061 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:54.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:54.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:54.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:54.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:54.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:54.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:54.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:54.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:54.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:54.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:54.944 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:54.945 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:54.945 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:55.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241898 2021-05-12T12:43:55.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241898 '!=' null 2021-05-12T12:43:55.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:55.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:55.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:55.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:55.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:55.541 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:55.541 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:55.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:55.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:55.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:55.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:55.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:55.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:55.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:55.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:56.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:56.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:56.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:56.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:56.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:56.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:56.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:56.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:56.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:56.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241898 2021-05-12T12:43:56.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241898 '!=' null 2021-05-12T12:43:56.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:56.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:56.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:57.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:57.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:57.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:57.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:57.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:57.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:57.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:57.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:57.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:57.415 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:57.416 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:57.416 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:57.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:57.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:57.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:57.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:57.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:57.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:57.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:57.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:57.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:58.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241898 2021-05-12T12:43:58.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241898 '!=' null 2021-05-12T12:43:58.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:58.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:58.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:58.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:58.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:58.567 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:58.567 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:58.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:58.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:58.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:58.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:58.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:58.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:58.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:58.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:59.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:59.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:59.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:59.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:59.424 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:59.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:59.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:59.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:59.426 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:59.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862917 2021-05-12T12:43:59.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862917 '!=' null 2021-05-12T12:43:59.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:59.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:59.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:00.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:00.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:00.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:00.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:00.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:00.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:00.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:00.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:00.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:00.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:00.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:00.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:00.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:00.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:00.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:00.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:00.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:00.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:00.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:00.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:00.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:01.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241901 2021-05-12T12:44:01.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241901 '!=' null 2021-05-12T12:44:01.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:01.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:01.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:01.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:01.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:01.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:01.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:01.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:01.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:01.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:01.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:01.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:01.826 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:01.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:01.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:02.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:02.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:02.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:02.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:02.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:02.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:02.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:02.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:02.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:02.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241901 2021-05-12T12:44:02.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241901 '!=' null 2021-05-12T12:44:02.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:02.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:02.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:02.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:02.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:02.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:02.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:02.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:02.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:02.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:02.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:03.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:03.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:03.264 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:03.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:03.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:03.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:03.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:03.759 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:03.759 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:03.759 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:03.759 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:03.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:03.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:04.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862847 2021-05-12T12:44:04.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862847 '!=' null 2021-05-12T12:44:04.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:04.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:04.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:04.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:04.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:04.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:04.466 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:04.466 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:04.466 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:04.466 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:04.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:04.822 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:04.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:04.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:04.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:05.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:05.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:05.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:05.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:05.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:05.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:05.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:05.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:05.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:05.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241940 2021-05-12T12:44:05.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241940 '!=' null 2021-05-12T12:44:05.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:05.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:05.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:05.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:05.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:05.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:05.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:05.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:05.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:05.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:05.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:06.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:06.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:06.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:06.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:06.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:06.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:06.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:06.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:06.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:06.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:06.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:06.783 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:06.783 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:07.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241940 2021-05-12T12:44:07.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241940 '!=' null 2021-05-12T12:44:07.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:07.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:07.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:07.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:07.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:07.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:07.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:07.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:07.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:07.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:07.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:07.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:07.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:07.734 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:07.734 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:08.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:08.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:08.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:08.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:08.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:08.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:08.230 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:08.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:08.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:08.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241998 2021-05-12T12:44:08.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241998 '!=' null 2021-05-12T12:44:08.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:08.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:08.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:08.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:08.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:08.837 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:08.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:08.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:08.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:08.839 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:08.839 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:09.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:09.196 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:09.196 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:09.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:09.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:09.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:09.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:09.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:09.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:09.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:09.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:09.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:09.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:10.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989347 2021-05-12T12:44:10.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989347 '!=' null 2021-05-12T12:44:10.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:10.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:10.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:10.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:10.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:10.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:10.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:10.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:10.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:10.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:10.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:10.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:10.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:10.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:10.648 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:11.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:11.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:11.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:11.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:11.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:11.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:11.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:11.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:11.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:11.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368338 2021-05-12T12:44:11.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368338 '!=' null 2021-05-12T12:44:11.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:11.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:11.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:11.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:11.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:11.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:11.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:11.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:11.768 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:11.768 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:11.768 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:12.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:12.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:12.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:12.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:12.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:12.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:12.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:12.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:12.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:12.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:12.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:12.613 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:12.613 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:13.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368338 2021-05-12T12:44:13.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368338 '!=' null 2021-05-12T12:44:13.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:13.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:13.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:13.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:13.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:13.218 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:13.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:13.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:13.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:13.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:13.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:13.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:13.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:13.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:13.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:14.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:14.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:14.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:14.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:14.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:14.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:14.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:14.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:14.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:14.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989334 2021-05-12T12:44:14.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989334 '!=' null 2021-05-12T12:44:14.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:14.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:14.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:14.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:14.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:14.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:14.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:14.748 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:14.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:14.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:14.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:15.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:15.108 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:15.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:15.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:15.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:15.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:15.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:15.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:15.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:15.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:15.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:15.606 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:15.607 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:16.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368332 2021-05-12T12:44:16.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368332 '!=' null 2021-05-12T12:44:16.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:16.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:16.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:16.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:16.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:16.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:16.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:16.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:16.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:16.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:16.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:16.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:16.572 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:16.572 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:16.573 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:17.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:17.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:17.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:17.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:17.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:17.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:17.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:17.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:17.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:17.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368286 2021-05-12T12:44:17.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368286 '!=' null 2021-05-12T12:44:17.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:17.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:17.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:17.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:17.654 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:17.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:17.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:17.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:17.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:17.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:17.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:18.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:18.016 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:18.016 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:18.016 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:18.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:18.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:18.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:18.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:18.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:18.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:18.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:18.500 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:18.501 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:19.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368286 2021-05-12T12:44:19.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368286 '!=' null 2021-05-12T12:44:19.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:19.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:19.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:19.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:19.140 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:19.141 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:19.141 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:19.141 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:19.142 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:19.142 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:19.142 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:19.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:19.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:19.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:19.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:20.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:20.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:20.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:20.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:20.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:20.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:20.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:20.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:20.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:20.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989183 2021-05-12T12:44:20.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989183 '!=' null 2021-05-12T12:44:20.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:20.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:20.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:20.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:20.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:20.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:20.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:20.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:20.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:20.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:20.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:20.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:20.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:20.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:20.976 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:21.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:21.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:21.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:21.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:21.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:21.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:21.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:21.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:21.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:21.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241874 2021-05-12T12:44:21.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241874 '!=' null 2021-05-12T12:44:21.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:21.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:21.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:22.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:22.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:22.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:22.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:22.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:22.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:22.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:22.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:22.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:22.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:22.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:22.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:22.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:22.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:22.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:22.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:22.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:22.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:22.940 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:22.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:22.941 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:23.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989162 2021-05-12T12:44:23.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989162 '!=' null 2021-05-12T12:44:23.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:23.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:23.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:23.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:23.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:23.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:23.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:23.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:23.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:23.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:23.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:23.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:23.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:23.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:23.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:24.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:24.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:24.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:24.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:24.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:24.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:24.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:24.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:24.415 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:24.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989162 2021-05-12T12:44:24.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989162 '!=' null 2021-05-12T12:44:24.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:24.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:24.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:25.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:25.007 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:25.008 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:25.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:25.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:25.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:25.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:25.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:25.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:25.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:25.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:25.373 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:25.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:25.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:25.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:25.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:25.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:25.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:25.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:25.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:25.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:26.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368238 2021-05-12T12:44:26.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368238 '!=' null 2021-05-12T12:44:26.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:26.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:26.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:26.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:26.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:26.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:26.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:26.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:26.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:26.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:26.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:26.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:26.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:26.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:26.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:27.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:27.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:27.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:27.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:27.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:27.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:27.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:27.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:27.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:27.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368238 2021-05-12T12:44:27.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368238 '!=' null 2021-05-12T12:44:27.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:27.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:27.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:27.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:27.941 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:27.942 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:27.942 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:27.942 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:27.943 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:27.943 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:27.943 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:28.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:28.300 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:28.300 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:28.301 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:28.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:28.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:28.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:28.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:28.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:28.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:28.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:28.784 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:28.784 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:29.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368238 2021-05-12T12:44:29.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368238 '!=' null 2021-05-12T12:44:29.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:29.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:29.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:29.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:29.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:29.407 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:29.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:29.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:29.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:29.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:29.409 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:29.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:29.765 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:29.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:29.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:30.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:30.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:30.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:30.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:30.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:30.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:30.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:30.247 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:30.247 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:30.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989263 2021-05-12T12:44:30.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989263 '!=' null 2021-05-12T12:44:30.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:30.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:30.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:30.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:30.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:30.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:30.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:30.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:30.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:30.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:30.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:31.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:31.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:31.244 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:31.244 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:31.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:31.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:31.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:31.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:31.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:31.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:31.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:31.738 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:31.738 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:32.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368332 2021-05-12T12:44:32.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368332 '!=' null 2021-05-12T12:44:32.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:32.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:32.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:32.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:32.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:32.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:32.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:32.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:32.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:32.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:32.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:32.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:32.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:32.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:32.689 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:33.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:33.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:33.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:33.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:33.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:33.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:33.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:33.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:33.175 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:33.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863000 2021-05-12T12:44:33.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863000 '!=' null 2021-05-12T12:44:33.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:33.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:33.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:33.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:33.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:33.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:33.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:33.793 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:33.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:33.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:33.794 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:34.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:34.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:34.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:34.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:34.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:34.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:34.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:34.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:34.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:34.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:34.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:34.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:34.646 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:35.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863000 2021-05-12T12:44:35.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863000 '!=' null 2021-05-12T12:44:35.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:35.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:35.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:35.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:35.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:35.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:35.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:35.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:35.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:35.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:35.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:35.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:35.590 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:35.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:35.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:36.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:36.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:36.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:36.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:36.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:36.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:36.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:36.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:36.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:36.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115621 2021-05-12T12:44:36.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115621 '!=' null 2021-05-12T12:44:36.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:36.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:36.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:36.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:36.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:36.678 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:36.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:36.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:36.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:36.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:36.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:37.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:37.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:37.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:37.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:37.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:37.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:37.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:37.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:37.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:37.519 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:37.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:37.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:37.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:38.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115677 2021-05-12T12:44:38.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115677 '!=' null 2021-05-12T12:44:38.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:38.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:38.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:38.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:38.101 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:38.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:38.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:38.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:38.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:38.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:38.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:38.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:38.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:38.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:38.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:38.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:38.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:38.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:38.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:38.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:38.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:38.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:38.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:38.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:39.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736678 2021-05-12T12:44:39.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736678 '!=' null 2021-05-12T12:44:39.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:39.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:39.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:39.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:39.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:39.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:39.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:39.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:39.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:39.537 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:39.537 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:39.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:39.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:39.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:39.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:40.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:40.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:40.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:40.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:40.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:40.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:40.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:40.371 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:40.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:40.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736678 2021-05-12T12:44:40.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736678 '!=' null 2021-05-12T12:44:40.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:40.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:40.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:40.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:40.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:40.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:40.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:40.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:40.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:40.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:40.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:41.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:41.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:41.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:41.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:41.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:41.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:41.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:41.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:41.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:41.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:41.803 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:41.803 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:41.804 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:42.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115654 2021-05-12T12:44:42.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115654 '!=' null 2021-05-12T12:44:42.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:42.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:42.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:42.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:42.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:42.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:42.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:42.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:42.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:42.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:42.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:42.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:42.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:42.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:42.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:43.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:43.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:43.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:43.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:43.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:43.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:43.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:43.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:43.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:43.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862978 2021-05-12T12:44:43.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862978 '!=' null 2021-05-12T12:44:43.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:43.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:43.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:43.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:43.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:43.842 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:43.842 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:43.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:43.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:43.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:43.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:44.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:44.202 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:44.202 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:44.202 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:44.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:44.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:44.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:44.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:44.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:44.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:44.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:44.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:44.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:45.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862978 2021-05-12T12:44:45.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862978 '!=' null 2021-05-12T12:44:45.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:45.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:45.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:45.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:45.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:45.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:45.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:45.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:45.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:45.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:45.272 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:45.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:45.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:45.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:45.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:46.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:46.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:46.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:46.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:46.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:46.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:46.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:46.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:46.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:46.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368307 2021-05-12T12:44:46.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368307 '!=' null 2021-05-12T12:44:46.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:46.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:46.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:46.685 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:46.686 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:46.687 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:46.687 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:46.687 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:46.687 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:46.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:46.688 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:47.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:47.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:47.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:47.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:47.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:47.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:47.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:47.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:47.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:47.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:47.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:47.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:47.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:48.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368350 2021-05-12T12:44:48.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368350 '!=' null 2021-05-12T12:44:48.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:48.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:48.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:48.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:48.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:48.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:48.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:48.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:48.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:48.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:48.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:48.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:48.491 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:48.491 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:48.491 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:48.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:48.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:48.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:48.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:48.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:48.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:48.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:48.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:48.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:49.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989346 2021-05-12T12:44:49.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989346 '!=' null 2021-05-12T12:44:49.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:49.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:49.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:49.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:49.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:49.601 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:49.601 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:49.601 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:49.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:49.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:49.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:49.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:49.963 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:49.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:49.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:50.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:50.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:50.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:50.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:50.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:50.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:50.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:50.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:50.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:50.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989346 2021-05-12T12:44:50.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989346 '!=' null 2021-05-12T12:44:50.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:50.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:50.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:51.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:51.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:51.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:51.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:51.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:51.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:51.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:51.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:51.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:51.430 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:51.430 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:51.430 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:51.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:51.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:51.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:51.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:51.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:51.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:51.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:51.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:51.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:52.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368347 2021-05-12T12:44:52.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368347 '!=' null 2021-05-12T12:44:52.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:52.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:52.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:52.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:52.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:52.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:52.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:52.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:52.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:52.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:52.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:52.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:52.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:52.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:52.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:53.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:53.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:53.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:53.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:53.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:53.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:53.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:53.474 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:53.474 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:53.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115699 2021-05-12T12:44:53.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115699 '!=' null 2021-05-12T12:44:53.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:53.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:53.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:54.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:54.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:54.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:54.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:54.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:54.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:54.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:54.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:54.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:54.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:54.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:54.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:54.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:54.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:54.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:54.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:54.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:54.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:54.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:54.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:54.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:55.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368352 2021-05-12T12:44:55.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368352 '!=' null 2021-05-12T12:44:55.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:55.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:55.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:55.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:55.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:55.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:55.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:55.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:55.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:55.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:55.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:55.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:55.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:55.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:55.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:56.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:56.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:56.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:56.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:56.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:56.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:56.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:56.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:56.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:56.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368352 2021-05-12T12:44:56.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368352 '!=' null 2021-05-12T12:44:56.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:56.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:56.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:56.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:56.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:56.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:56.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:56.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:56.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:56.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:56.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:57.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:57.328 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:57.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:57.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:57.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:57.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:57.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:57.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:57.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:57.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:57.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:57.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:57.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:58.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368374 2021-05-12T12:44:58.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368374 '!=' null 2021-05-12T12:44:58.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:58.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:58.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:58.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:58.442 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:58.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:58.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:58.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:58.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:58.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:58.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:58.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:58.804 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:58.804 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:58.804 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:59.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:59.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:59.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:59.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:59.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:59.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:59.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:59.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:59.300 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:59.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989377 2021-05-12T12:44:59.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989377 '!=' null 2021-05-12T12:44:59.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:59.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:59.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:59.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:59.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:59.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:59.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:59.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:59.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:59.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:59.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:00.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:00.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:00.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:00.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:00.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:00.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:00.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:00.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:00.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:00.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:00.755 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:00.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:00.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:01.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989377 2021-05-12T12:45:01.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989377 '!=' null 2021-05-12T12:45:01.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:01.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:01.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:01.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:01.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:01.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:01.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:01.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:01.341 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:01.342 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:01.342 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:01.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:01.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:01.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:01.707 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:02.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:02.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:02.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:02.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:02.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:02.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:02.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:02.215 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:02.216 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:02.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368364 2021-05-12T12:45:02.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368364 '!=' null 2021-05-12T12:45:02.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:02.709 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:02.710 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:02.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:02.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:02.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:02.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:02.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:02.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:02.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:02.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:03.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:03.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:03.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:03.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:03.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:03.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:03.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:03.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:03.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:03.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:03.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:03.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:03.658 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:04.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989368 2021-05-12T12:45:04.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989368 '!=' null 2021-05-12T12:45:04.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:04.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:04.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:04.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:04.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:04.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:04.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:04.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:04.248 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:04.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:04.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:04.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:04.608 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:04.608 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:04.608 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:05.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:05.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:05.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:05.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:05.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:05.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:05.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:05.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:05.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:05.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241998 2021-05-12T12:45:05.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241998 '!=' null 2021-05-12T12:45:05.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:05.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:05.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:05.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:05.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:05.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:05.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:05.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:05.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:05.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:05.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:06.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:06.042 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:06.042 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:06.043 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:06.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:06.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:06.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:06.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:06.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:06.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:06.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:06.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:06.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:07.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241998 2021-05-12T12:45:07.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241998 '!=' null 2021-05-12T12:45:07.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:07.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:07.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:07.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:07.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:07.129 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:07.129 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:07.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:07.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:07.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:07.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:07.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:07.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:07.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:07.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:08.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:08.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:08.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:08.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:08.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:08.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:08.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:08.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:08.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:08.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241976 2021-05-12T12:45:08.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241976 '!=' null 2021-05-12T12:45:08.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:08.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:08.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:08.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:08.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:08.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:08.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:08.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:08.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:08.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:08.621 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:08.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:08.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:08.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:08.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:09.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:09.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:09.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:09.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:09.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:09.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:09.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:09.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:09.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:09.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862950 2021-05-12T12:45:09.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862950 '!=' null 2021-05-12T12:45:09.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:09.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:09.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:10.048 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:10.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:10.050 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:10.050 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:10.050 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:10.050 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:10.051 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:10.051 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:10.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:10.413 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:10.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:10.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:10.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:10.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:10.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:10.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:10.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:10.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:10.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:10.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:10.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:11.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862950 2021-05-12T12:45:11.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862950 '!=' null 2021-05-12T12:45:11.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:11.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:11.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:11.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:11.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:11.500 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:11.500 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:11.500 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:11.501 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:11.501 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:11.501 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:11.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:11.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:11.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:11.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:12.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:12.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:12.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:12.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:12.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:12.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:12.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:12.362 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:12.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:12.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241963 2021-05-12T12:45:12.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241963 '!=' null 2021-05-12T12:45:12.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:12.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:12.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:12.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:12.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:12.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:12.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:12.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:12.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:12.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:12.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:13.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:13.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:13.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:13.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:13.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:13.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:13.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:13.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:13.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:13.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:13.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:13.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:13.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:14.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862949 2021-05-12T12:45:14.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862949 '!=' null 2021-05-12T12:45:14.335 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:14.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:14.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:14.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:14.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:14.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:14.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:14.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:14.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:14.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:14.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:14.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:14.799 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:14.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:14.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:15.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:15.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:15.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:15.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:15.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:15.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:15.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:15.288 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:15.289 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:15.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241953 2021-05-12T12:45:15.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241953 '!=' null 2021-05-12T12:45:15.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:15.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:15.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:15.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:15.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:15.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:15.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:15.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:15.915 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:15.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:15.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:16.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:16.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:16.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:16.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:16.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:16.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:16.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:16.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:16.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:16.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:16.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:16.774 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:16.774 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:17.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241953 2021-05-12T12:45:17.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241953 '!=' null 2021-05-12T12:45:17.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:17.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:17.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:17.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:17.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:17.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:17.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:17.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:17.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:17.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:17.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:17.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:17.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:17.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:17.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:18.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:18.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:18.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:18.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:18.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:18.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:18.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:18.214 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:18.214 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:18.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241964 2021-05-12T12:45:18.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241964 '!=' null 2021-05-12T12:45:18.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:18.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:18.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:18.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:18.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:18.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:18.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:18.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:18.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:18.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:18.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:19.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:19.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:19.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:19.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:19.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:19.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:19.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:19.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:19.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:19.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:19.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:19.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:19.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:20.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862973 2021-05-12T12:45:20.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862973 '!=' null 2021-05-12T12:45:20.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:20.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:20.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:20.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:20.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:20.273 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:20.273 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:20.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:20.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:20.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:20.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:20.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:20.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:20.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:20.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:21.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:21.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:21.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:21.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:21.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:21.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:21.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:21.123 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:21.123 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:21.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241980 2021-05-12T12:45:21.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241980 '!=' null 2021-05-12T12:45:21.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:21.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:21.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:21.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:21.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:21.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:21.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:21.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:21.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:21.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:21.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:22.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:22.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:22.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:22.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:22.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:22.555 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:22.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:22.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:22.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:22.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:22.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:22.557 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:22.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:23.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241980 2021-05-12T12:45:23.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241980 '!=' null 2021-05-12T12:45:23.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:23.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:23.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:23.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:23.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:23.146 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:23.146 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:23.146 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:23.146 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:23.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:23.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:23.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:23.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:23.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:23.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:24.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:24.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:24.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:24.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:24.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:24.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:24.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:24.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:24.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:24.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862973 2021-05-12T12:45:24.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862973 '!=' null 2021-05-12T12:45:24.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:24.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:24.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:24.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:24.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:24.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:24.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:24.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:24.632 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:24.633 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:24.633 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:24.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:24.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:24.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:24.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:25.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:25.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:25.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:25.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:25.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:25.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:25.499 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:25.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:25.500 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:25.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241963 2021-05-12T12:45:25.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241963 '!=' null 2021-05-12T12:45:25.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:25.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:25.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:26.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:26.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:26.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:26.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:26.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:26.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:26.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:26.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:26.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:26.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:26.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:26.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:26.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:26.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:26.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:26.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:26.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:26.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:26.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:26.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:26.926 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:27.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241947 2021-05-12T12:45:27.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241947 '!=' null 2021-05-12T12:45:27.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:27.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:27.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:27.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:27.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:27.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:27.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:27.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:27.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:27.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:27.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:27.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:27.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:27.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:27.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:28.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:28.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:28.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:28.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:28.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:28.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:28.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:28.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:28.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:28.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241947 2021-05-12T12:45:28.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241947 '!=' null 2021-05-12T12:45:28.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:28.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:28.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:28.983 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:28.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:28.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:28.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:28.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:28.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:28.986 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:28.986 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:29.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:29.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:29.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:29.345 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:29.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:29.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:29.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:29.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:29.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:29.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:29.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:29.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:29.890 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:30.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989228 2021-05-12T12:45:30.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989228 '!=' null 2021-05-12T12:45:30.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:30.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:30.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:30.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:30.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:30.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:30.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:30.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:30.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:30.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:30.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:30.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:30.847 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:30.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:30.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:31.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:31.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:31.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:31.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:31.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:31.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:31.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:31.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:31.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:31.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368262 2021-05-12T12:45:31.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368262 '!=' null 2021-05-12T12:45:31.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:31.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:31.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:31.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:31.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:31.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:31.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:31.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:31.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:31.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:31.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:32.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:32.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:32.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:32.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:32.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:32.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:32.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:32.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:32.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:32.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:32.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:32.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:32.790 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:33.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368262 2021-05-12T12:45:33.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368262 '!=' null 2021-05-12T12:45:33.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:33.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:33.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:33.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:33.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:33.380 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:33.380 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:33.380 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:33.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:33.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:33.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:33.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:33.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:33.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:33.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:34.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:34.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:34.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:34.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:34.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:34.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:34.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:34.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:34.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:34.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989228 2021-05-12T12:45:34.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989228 '!=' null 2021-05-12T12:45:34.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:34.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:34.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:34.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:34.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:34.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:34.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:34.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:34.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:34.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:34.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:35.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:35.179 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:35.179 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:35.179 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:35.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:35.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:35.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:35.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:35.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:35.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:35.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:35.696 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:35.696 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:36.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368279 2021-05-12T12:45:36.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368279 '!=' null 2021-05-12T12:45:36.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:36.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:36.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:36.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:36.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:36.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:36.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:36.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:36.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:36.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:36.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:36.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:36.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:36.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:36.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:37.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:37.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:37.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:37.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:37.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:37.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:37.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:37.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:37.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:37.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368248 2021-05-12T12:45:37.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368248 '!=' null 2021-05-12T12:45:37.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:37.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:37.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:37.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:37.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:37.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:37.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:37.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:37.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:37.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:37.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:38.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:38.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:38.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:38.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:38.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:38.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:38.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:38.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:38.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:38.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:38.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:38.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:38.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:39.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368248 2021-05-12T12:45:39.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368248 '!=' null 2021-05-12T12:45:39.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:39.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:39.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:39.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:39.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:39.222 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:39.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:39.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:39.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:39.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:39.224 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:39.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:39.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:39.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:39.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:40.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:40.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:40.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:40.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:40.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:40.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:40.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:40.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:40.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:40.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989270 2021-05-12T12:45:40.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989270 '!=' null 2021-05-12T12:45:40.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:40.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:40.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:40.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:40.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:40.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:40.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:40.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:40.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:40.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:40.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:41.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:41.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:41.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:41.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:41.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:41.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:41.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:41.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:41.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:41.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:41.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:41.547 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:41.547 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:42.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241934 2021-05-12T12:45:42.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241934 '!=' null 2021-05-12T12:45:42.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:42.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:42.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:42.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:42.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:42.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:42.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:42.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:42.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:42.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:42.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:42.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:42.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:42.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:42.494 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:42.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:42.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:42.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:42.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:42.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:42.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:42.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:42.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:42.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:43.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989262 2021-05-12T12:45:43.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989262 '!=' null 2021-05-12T12:45:43.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:43.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:43.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:43.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:43.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:43.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:43.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:43.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:43.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:43.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:43.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:43.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:43.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:43.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:43.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:44.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:44.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:44.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:44.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:44.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:44.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:44.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:44.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:44.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:44.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989262 2021-05-12T12:45:44.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989262 '!=' null 2021-05-12T12:45:44.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:44.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:44.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:45.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:45.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:45.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:45.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:45.032 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:45.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:45.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:45.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:45.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:45.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:45.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:45.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:45.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:45.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:45.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:45.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:45.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:45.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:45.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:45.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:45.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:46.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368288 2021-05-12T12:45:46.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368288 '!=' null 2021-05-12T12:45:46.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:46.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:46.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:46.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:46.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:46.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:46.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:46.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:46.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:46.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:46.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:46.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:46.848 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:46.849 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:46.849 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:47.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:47.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:47.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:47.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:47.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:47.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:47.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:47.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:47.327 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:47.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368240 2021-05-12T12:45:47.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368240 '!=' null 2021-05-12T12:45:47.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:47.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:47.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:48.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:48.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:48.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:48.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:48.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:48.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:48.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:48.041 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:48.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:48.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:48.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:48.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:48.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:48.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:48.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:48.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:48.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:48.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:48.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:48.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:48.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:49.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368240 2021-05-12T12:45:49.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368240 '!=' null 2021-05-12T12:45:49.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:49.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:49.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:49.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:49.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:49.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:49.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:49.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:49.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:49.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:49.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:49.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:49.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:49.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:49.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:50.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:50.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:50.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:50.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:50.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:50.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:50.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:50.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:50.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:50.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989255 2021-05-12T12:45:50.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989255 '!=' null 2021-05-12T12:45:50.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:50.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:50.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:50.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:50.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:50.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:50.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:50.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:50.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:50.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:50.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:51.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:51.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:51.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:51.272 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:51.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:51.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:51.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:51.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:51.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:51.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:51.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:51.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:51.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:52.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368267 2021-05-12T12:45:52.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368267 '!=' null 2021-05-12T12:45:52.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:52.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:52.261 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:52.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:52.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:52.362 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:52.362 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:52.362 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:52.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:52.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:52.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:52.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:52.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:52.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:52.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:53.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:53.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:53.197 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:53.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:53.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:53.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:53.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:53.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:53.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:53.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862973 2021-05-12T12:45:53.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862973 '!=' null 2021-05-12T12:45:53.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:53.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:53.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:53.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:53.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:53.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:53.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:53.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:53.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:53.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:53.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:54.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:54.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:54.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:54.132 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:54.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:54.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:54.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:54.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:54.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:54.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:54.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:54.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:54.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:55.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862973 2021-05-12T12:45:55.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862973 '!=' null 2021-05-12T12:45:55.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:55.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:55.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:55.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:55.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:55.230 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:55.230 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:55.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:55.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:55.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:55.231 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:55.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:55.633 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:55.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:55.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:56.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:56.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:56.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:56.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:56.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:56.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:56.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:56.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:56.123 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:56.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115653 2021-05-12T12:45:56.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115653 '!=' null 2021-05-12T12:45:56.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:56.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:56.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:56.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:56.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:56.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:56.715 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:56.716 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:56.716 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:56.716 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:56.716 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:57.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:57.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:57.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:57.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:57.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:57.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:57.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:57.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:57.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:57.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:57.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:57.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:57.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:58.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115648 2021-05-12T12:45:58.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115648 '!=' null 2021-05-12T12:45:58.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:58.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:58.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:58.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:58.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:58.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:58.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:58.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:58.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:58.281 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:58.281 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:58.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:58.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:58.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:58.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:59.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:59.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:59.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:59.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:59.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:59.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:59.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:59.126 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:59.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:59.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736689 2021-05-12T12:45:59.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736689 '!=' null 2021-05-12T12:45:59.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:59.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:59.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:59.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:59.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:59.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:59.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:59.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:59.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:59.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:59.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:00.074 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:00.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:00.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:00.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:00.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:00.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:00.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:00.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:00.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:00.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:00.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:00.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:00.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:01.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736689 2021-05-12T12:46:01.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736689 '!=' null 2021-05-12T12:46:01.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:01.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:01.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:01.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:01.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:01.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:01.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:01.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:01.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:01.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:01.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:01.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:01.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:01.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:01.520 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:02.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:02.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:02.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:02.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:02.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:02.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:02.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:02.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:02.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:02.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115673 2021-05-12T12:46:02.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115673 '!=' null 2021-05-12T12:46:02.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:02.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:02.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:02.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:02.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:02.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:02.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:02.611 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:02.611 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:02.611 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:02.611 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:02.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:02.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:02.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:02.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:03.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:03.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:03.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:03.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:03.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:03.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:03.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:03.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:03.450 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:03.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863033 2021-05-12T12:46:03.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863033 '!=' null 2021-05-12T12:46:03.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:03.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:03.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:04.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:04.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:04.086 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:04.086 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:04.086 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:04.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:04.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:04.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:04.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:04.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:04.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:04.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:04.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:04.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:04.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:04.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:04.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:04.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:04.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:04.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:04.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:05.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368344 2021-05-12T12:46:05.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368344 '!=' null 2021-05-12T12:46:05.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:05.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:05.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:05.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:05.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:05.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:05.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:05.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:05.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:05.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:05.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:05.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:05.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:05.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:05.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:06.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:06.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:06.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:06.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:06.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:06.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:06.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:06.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:06.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:06.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368344 2021-05-12T12:46:06.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368344 '!=' null 2021-05-12T12:46:06.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:06.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:06.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:06.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:06.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:06.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:06.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:06.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:06.963 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:06.963 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:06.963 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:07.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:07.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:07.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:07.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:07.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:07.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:07.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:07.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:07.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:07.806 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:07.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:07.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:07.808 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:08.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368332 2021-05-12T12:46:08.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368332 '!=' null 2021-05-12T12:46:08.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:08.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:08.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:08.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:08.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:08.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:08.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:08.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:08.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:08.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:08.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:08.774 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:08.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:08.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:08.775 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:09.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:09.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:09.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:09.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:09.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:09.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:09.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:09.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:09.257 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:09.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989317 2021-05-12T12:46:09.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989317 '!=' null 2021-05-12T12:46:09.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:09.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:09.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:09.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:09.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:09.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:09.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:09.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:09.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:09.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:09.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:10.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:10.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:10.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:10.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:10.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:10.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:10.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:10.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:10.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:10.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:10.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:10.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:10.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:11.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989317 2021-05-12T12:46:11.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989317 '!=' null 2021-05-12T12:46:11.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:11.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:11.276 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:11.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:11.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:11.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:11.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:11.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:11.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:11.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:11.378 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:11.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:11.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:11.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:11.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:12.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:12.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:12.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:12.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:12.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:12.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:12.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:12.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:12.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:12.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368334 2021-05-12T12:46:12.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368334 '!=' null 2021-05-12T12:46:12.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:12.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:12.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:12.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:12.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:12.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:12.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:12.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:12.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:12.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:12.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:13.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:13.171 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:13.171 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:13.172 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:13.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:13.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:13.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:13.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:13.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:13.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:13.689 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:13.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:13.690 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:14.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241982 2021-05-12T12:46:14.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241982 '!=' null 2021-05-12T12:46:14.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:14.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:14.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:14.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:14.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:14.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:14.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:14.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:14.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:14.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:14.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:14.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:14.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:14.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:14.640 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:15.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:15.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:15.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:15.140 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:15.140 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:15.140 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:15.140 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:15.141 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:15.141 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:15.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241975 2021-05-12T12:46:15.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241975 '!=' null 2021-05-12T12:46:15.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:15.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:15.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:15.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:15.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:15.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:15.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:15.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:15.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:15.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:15.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:16.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:16.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:16.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:16.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:16.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:16.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:16.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:16.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:16.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:16.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:16.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:16.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:16.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:17.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241975 2021-05-12T12:46:17.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241975 '!=' null 2021-05-12T12:46:17.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:17.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:17.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:17.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:17.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:17.170 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:17.171 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:17.171 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:17.171 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:17.171 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:17.172 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:17.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:17.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:17.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:17.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:18.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:18.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:18.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:18.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:18.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:18.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:18.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:18.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:18.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:18.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241963 2021-05-12T12:46:18.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241963 '!=' null 2021-05-12T12:46:18.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:18.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:18.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:18.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:18.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:18.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:18.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:18.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:18.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:18.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:18.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:18.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:18.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:18.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:18.985 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:19.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:19.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:19.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:19.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:19.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:19.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:19.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:19.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:19.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:20.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989297 2021-05-12T12:46:20.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989297 '!=' null 2021-05-12T12:46:20.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:20.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:20.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:20.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:20.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:20.145 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:20.145 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:20.145 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:20.145 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:20.146 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:20.146 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:20.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:20.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:20.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:20.502 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:20.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:20.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:20.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:20.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:20.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:20.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:21.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T12:46:21.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T12:46:21.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:21.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:21.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:21.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:21.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:21.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:21.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:21.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:21.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:21.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:21.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:21.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:21.987 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:21.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:21.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:22.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:22.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:22.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:22.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:22.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:22.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:22.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:22.479 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:22.479 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:22.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T12:46:22.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T12:46:22.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:22.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:22.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:23.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:23.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:23.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:23.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:23.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:23.088 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:23.088 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:23.088 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:23.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:23.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:23.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:23.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:23.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:23.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:23.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:23.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:23.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:23.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:23.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:23.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:23.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:24.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241991 2021-05-12T12:46:24.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241991 '!=' null 2021-05-12T12:46:24.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:24.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:24.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:24.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:24.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:24.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:24.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:24.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:24.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:24.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:24.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:24.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:24.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:24.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:24.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:25.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:25.378 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:25.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:25.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:25.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:25.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:25.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:25.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:25.381 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:25.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115685 2021-05-12T12:46:25.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115685 '!=' null 2021-05-12T12:46:25.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:25.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:25.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:26.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:26.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:26.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:26.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:26.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:26.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:26.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:26.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:26.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:26.390 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:26.391 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:26.391 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:26.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:26.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:26.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:26.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:26.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:26.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:26.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:26.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:26.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:27.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368345 2021-05-12T12:46:27.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368345 '!=' null 2021-05-12T12:46:27.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:27.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:27.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:27.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:27.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:27.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:27.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:27.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:27.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:27.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:27.505 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:27.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:27.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:27.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:27.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:28.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:28.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:28.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:28.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:28.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:28.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:28.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:28.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:28.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:28.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368345 2021-05-12T12:46:28.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368345 '!=' null 2021-05-12T12:46:28.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:28.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:28.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:29.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:29.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:29.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:29.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:29.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:29.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:29.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:29.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:29.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:29.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:29.360 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:29.360 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:29.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:29.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:29.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:29.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:29.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:29.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:29.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:29.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:29.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:30.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989330 2021-05-12T12:46:30.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989330 '!=' null 2021-05-12T12:46:30.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:30.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:30.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:30.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:30.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:30.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:30.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:30.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:30.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:30.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:30.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:30.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:30.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:30.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:30.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:31.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:31.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:31.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:31.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:31.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:31.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:31.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:31.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:31.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:31.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241961 2021-05-12T12:46:31.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241961 '!=' null 2021-05-12T12:46:31.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:31.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:31.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:31.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:31.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:31.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:31.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:31.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:31.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:31.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:31.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:32.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:32.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:32.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:32.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:32.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:32.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:32.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:32.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:32.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:32.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:32.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:32.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:32.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:33.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241961 2021-05-12T12:46:33.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241961 '!=' null 2021-05-12T12:46:33.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:33.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:33.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:33.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:33.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:33.308 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:33.308 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:33.308 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:33.308 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:33.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:33.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:33.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:33.664 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:33.665 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:33.665 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:34.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:34.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:34.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:34.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:34.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:34.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:34.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:34.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:34.150 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:34.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989319 2021-05-12T12:46:34.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989319 '!=' null 2021-05-12T12:46:34.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:34.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:34.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:34.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:34.735 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:34.736 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:34.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:34.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:34.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:34.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:34.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:35.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:35.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:35.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:35.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:35.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:35.575 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:35.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:35.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:35.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:35.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:35.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:35.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:35.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:36.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862958 2021-05-12T12:46:36.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862958 '!=' null 2021-05-12T12:46:36.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:36.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:36.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:36.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:36.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:36.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:36.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:36.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:36.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:36.184 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:36.184 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:36.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:36.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:36.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:36.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:37.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:37.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:37.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:37.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:37.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:37.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:37.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:37.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:37.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:37.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115602 2021-05-12T12:46:37.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115602 '!=' null 2021-05-12T12:46:37.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:37.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:37.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:37.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:37.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:37.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:37.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:37.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:37.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:37.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:37.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:38.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:38.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:38.037 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:38.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:38.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:38.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:38.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:38.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:38.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:38.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:38.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:38.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:38.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:39.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115602 2021-05-12T12:46:39.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115602 '!=' null 2021-05-12T12:46:39.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:39.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:39.015 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:39.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:39.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:39.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:39.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:39.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:39.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:39.117 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:39.117 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:39.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:39.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:39.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:39.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:39.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:39.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:39.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:39.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:39.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:39.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:39.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:39.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:39.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:40.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862834 2021-05-12T12:46:40.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862834 '!=' null 2021-05-12T12:46:40.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:40.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:40.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:40.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:40.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:40.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:40.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:40.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:40.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:40.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:40.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:40.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:40.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:40.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:40.928 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:41.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:41.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:41.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:41.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:41.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:41.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:41.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:41.405 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:41.405 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:41.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241916 2021-05-12T12:46:41.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241916 '!=' null 2021-05-12T12:46:41.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:41.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:41.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:42.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:42.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:42.054 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:42.054 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:42.054 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:42.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:42.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:42.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:42.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:42.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:42.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:42.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:42.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:42.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:42.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:42.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:42.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:42.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:42.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:42.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:42.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:43.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862865 2021-05-12T12:46:43.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862865 '!=' null 2021-05-12T12:46:43.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:43.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:43.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:43.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:43.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:43.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:43.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:43.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:43.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:43.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:43.504 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:43.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:43.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:43.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:43.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:44.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:44.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:44.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:44.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:44.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:44.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:44.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:44.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:44.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:44.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862865 2021-05-12T12:46:44.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862865 '!=' null 2021-05-12T12:46:44.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:44.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:44.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:45.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:45.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:45.022 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:45.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:45.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:45.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:45.023 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:45.024 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:45.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:45.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:45.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:45.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:45.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:45.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:45.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:45.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:45.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:45.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:45.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:45.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:45.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:46.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115543 2021-05-12T12:46:46.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115543 '!=' null 2021-05-12T12:46:46.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:46.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:46.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:46.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:46.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:46.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:46.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:46.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:46.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:46.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:46.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:46.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:46.864 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:46.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:46.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:47.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:47.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:47.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:47.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:47.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:47.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:47.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:47.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:47.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:47.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368221 2021-05-12T12:46:47.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368221 '!=' null 2021-05-12T12:46:47.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:47.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:47.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:47.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:47.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:47.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:47.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:47.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:47.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:47.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:47.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:48.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:48.318 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:48.318 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:48.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:48.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:48.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:48.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:48.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:48.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:48.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:48.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:48.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:48.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:49.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368221 2021-05-12T12:46:49.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368221 '!=' null 2021-05-12T12:46:49.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:49.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:49.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:49.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:49.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:49.395 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:49.396 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:49.396 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:49.396 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:49.396 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:49.397 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:49.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:49.756 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:49.757 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:49.757 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:50.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:50.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:50.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:50.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:50.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:50.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:50.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:50.247 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:50.247 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:50.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115504 2021-05-12T12:46:50.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115504 '!=' null 2021-05-12T12:46:50.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:50.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:50.743 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:50.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:50.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:50.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:50.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:50.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:50.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:50.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:50.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:51.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:51.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:51.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:51.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:51.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:51.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:51.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:51.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:51.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:51.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:51.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:51.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:51.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:52.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368230 2021-05-12T12:46:52.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368230 '!=' null 2021-05-12T12:46:52.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:52.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:52.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:52.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:52.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:52.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:52.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:52.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:52.288 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:52.288 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:52.288 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:52.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:52.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:52.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:52.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:53.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:53.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:53.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:53.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:53.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:53.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:53.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:53.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:53.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:53.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989094 2021-05-12T12:46:53.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989094 '!=' null 2021-05-12T12:46:53.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:53.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:53.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:53.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:53.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:53.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:53.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:53.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:53.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:53.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:53.734 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:54.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:54.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:54.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:54.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:54.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:54.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:54.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:54.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:54.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:54.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:54.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:54.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:54.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:55.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989094 2021-05-12T12:46:55.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989094 '!=' null 2021-05-12T12:46:55.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:55.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:55.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:55.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:55.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:55.173 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:55.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:55.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:55.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:55.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:55.175 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:55.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:55.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:55.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:55.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:56.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:56.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:56.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:56.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:56.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:56.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:56.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:56.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:56.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:56.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368230 2021-05-12T12:46:56.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368230 '!=' null 2021-05-12T12:46:56.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:56.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:56.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:56.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:56.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:56.622 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:56.622 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:56.622 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:56.622 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:56.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:56.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:56.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:56.982 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:56.983 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:56.983 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:57.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:57.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:57.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:57.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:57.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:57.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:57.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:57.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:57.721 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:58.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368233 2021-05-12T12:46:58.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368233 '!=' null 2021-05-12T12:46:58.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:58.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:58.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:58.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:58.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:58.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:58.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:58.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:58.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:58.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:58.308 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:58.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:46:58.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:58.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:58.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:59.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:46:59.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:46:59.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:59.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:59.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:59.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:59.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:59.153 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:59.154 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:59.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989157 2021-05-12T12:46:59.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989157 '!=' null 2021-05-12T12:46:59.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:59.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:59.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:59.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:59.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:59.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:59.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:59.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:59.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:59.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:59.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:00.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:00.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:00.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:00.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:00.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:00.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:00.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:00.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:00.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:00.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:00.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:00.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:00.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:01.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989157 2021-05-12T12:47:01.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989157 '!=' null 2021-05-12T12:47:01.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:01.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:01.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:01.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:01.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:01.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:01.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:01.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:01.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:01.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:01.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:01.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:01.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:01.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:01.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:02.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:02.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:02.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:02.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:02.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:02.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:02.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:02.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:02.149 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:02.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241885 2021-05-12T12:47:02.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241885 '!=' null 2021-05-12T12:47:02.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:02.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:02.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:02.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:02.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:02.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:02.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:02.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:02.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:02.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:02.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:03.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:03.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:03.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:03.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:03.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:03.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:03.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:03.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:03.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:03.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:03.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:03.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:03.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:04.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989215 2021-05-12T12:47:04.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989215 '!=' null 2021-05-12T12:47:04.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:04.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:04.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:04.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:04.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:04.184 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:04.184 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:04.184 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:04.185 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:04.185 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:04.185 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:04.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:04.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:04.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:04.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:05.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:05.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:05.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:05.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:05.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:05.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:05.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:05.025 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:05.026 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:05.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368311 2021-05-12T12:47:05.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368311 '!=' null 2021-05-12T12:47:05.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:05.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:05.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:05.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:05.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:05.609 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:05.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:05.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:05.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:05.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:05.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:05.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:05.966 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:05.966 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:05.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:06.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:06.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:06.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:06.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:06.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:06.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:06.462 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:06.462 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:06.463 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:07.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368311 2021-05-12T12:47:07.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368311 '!=' null 2021-05-12T12:47:07.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:07.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:07.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:07.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:07.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:07.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:07.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:07.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:07.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:07.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:07.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:07.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:07.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:07.466 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:07.466 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:07.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:07.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:07.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:07.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:07.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:07.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:07.953 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:07.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:07.954 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:08.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368306 2021-05-12T12:47:08.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368306 '!=' null 2021-05-12T12:47:08.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:08.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:08.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:08.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:08.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:08.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:08.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:08.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:08.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:08.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:08.540 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:08.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:08.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:08.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:08.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:09.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:09.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:09.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:09.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:09.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:09.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:09.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:09.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:09.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:09.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989297 2021-05-12T12:47:09.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989297 '!=' null 2021-05-12T12:47:09.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:09.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:09.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:10.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:10.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:10.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:10.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:10.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:10.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:10.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:10.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:10.352 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:10.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:10.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:10.354 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:10.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:10.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:10.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:10.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:10.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:10.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:10.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:10.862 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:10.862 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:11.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989297 2021-05-12T12:47:11.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989297 '!=' null 2021-05-12T12:47:11.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:11.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:11.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:11.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:11.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:11.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:11.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:11.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:11.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:11.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:11.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:11.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:11.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:11.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:11.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:12.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:12.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:12.301 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:12.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:12.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:12.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:12.302 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:12.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:12.303 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:12.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368319 2021-05-12T12:47:12.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368319 '!=' null 2021-05-12T12:47:12.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:12.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:12.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:12.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:12.884 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:12.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:12.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:12.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:12.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:12.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:12.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:13.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:13.246 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:13.246 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:13.246 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:13.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:13.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:13.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:13.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:13.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:13.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:13.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:13.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:13.733 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:14.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863006 2021-05-12T12:47:14.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863006 '!=' null 2021-05-12T12:47:14.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:14.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:14.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:14.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:14.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:14.325 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:14.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:14.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:14.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:14.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:14.327 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:14.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:14.685 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:14.685 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:14.685 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:15.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:15.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:15.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:15.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:15.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:15.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:15.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:15.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:15.175 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:15.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115664 2021-05-12T12:47:15.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115664 '!=' null 2021-05-12T12:47:15.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:15.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:15.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:15.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:15.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:15.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:15.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:15.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:15.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:15.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:15.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:16.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:16.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:16.199 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:16.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:16.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:16.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:16.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:16.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:16.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:16.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:16.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:16.699 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:16.699 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:17.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115664 2021-05-12T12:47:17.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115664 '!=' null 2021-05-12T12:47:17.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:17.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:17.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:17.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:17.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:17.311 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:17.311 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:17.311 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:17.311 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:17.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:17.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:17.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:17.684 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:17.684 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:17.684 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:18.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:18.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:18.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:18.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:18.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:18.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:18.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:18.179 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:18.179 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:18.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7338759 2021-05-12T12:47:18.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7338759 '!=' null 2021-05-12T12:47:18.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:18.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:18.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:18.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:18.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:18.765 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:18.765 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:18.765 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:18.765 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:18.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:18.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:19.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:47:19.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:19.118 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:19.119 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:19.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:47:19.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:47:19.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:19.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:19.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:19.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:19.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:19.596 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:19.596 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:20.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=9173518 2021-05-12T12:47:20.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 9173518 '!=' null 2021-05-12T12:47:20.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:20.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:20.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:20.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:20.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:20.168 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:20.168 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:20.168 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:20.168 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:20.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:20.169 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:20.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:20.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:20.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:20.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 0 2021-05-12T12:47:20.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:47:21.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=1 2021-05-12T12:47:21.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:21.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:21.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:21.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:21.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:21.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:21.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:21.101 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:21.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:21.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:21.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:21.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:21.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:21.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:22.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:22.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:22.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:22.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:22.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:22.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:22.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:22.062 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:22.062 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:22.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3931495 2021-05-12T12:47:22.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3931495 '!=' null 2021-05-12T12:47:22.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:22.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:22.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:22.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:22.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:22.640 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:22.640 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:22.641 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:22.641 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:22.641 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:22.641 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:22.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:22.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:22.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:22.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:23.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:23.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:23.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:23.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:23.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:23.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:23.489 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:23.489 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:23.490 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:23.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242052 2021-05-12T12:47:23.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242052 '!=' null 2021-05-12T12:47:23.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:23.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:23.982 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:24.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:24.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:24.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:24.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:24.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:24.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:24.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:24.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:24.434 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:24.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:24.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:24.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:24.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:24.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:24.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:24.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:24.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:24.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:24.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:24.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:24.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:25.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5169836 2021-05-12T12:47:25.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5169836 '!=' null 2021-05-12T12:47:25.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:25.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:25.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:25.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:25.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:25.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:25.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:25.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:25.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:25.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:25.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:25.942 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:25.943 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:25.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:25.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:26.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:26.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:26.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:26.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:26.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:26.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:26.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:26.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:26.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:26.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5169836 2021-05-12T12:47:26.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5169836 '!=' null 2021-05-12T12:47:26.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:26.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:26.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:27.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:27.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:27.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:27.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:27.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:27.019 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:27.019 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:27.019 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:27.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:27.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:27.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:27.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:27.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:27.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:27.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:27.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:27.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:27.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:27.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:27.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:27.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:28.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4193638 2021-05-12T12:47:28.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4193638 '!=' null 2021-05-12T12:47:28.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:28.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:28.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:28.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:28.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:28.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:28.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:28.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:28.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:28.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:28.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:28.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:28.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:28.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:28.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:29.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:29.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:29.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:29.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:29.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:29.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:29.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:29.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:29.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:29.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4467244 2021-05-12T12:47:29.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4467244 '!=' null 2021-05-12T12:47:29.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:29.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:29.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:29.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:29.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:29.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:29.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:29.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:29.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:29.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:29.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:30.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:30.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:30.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:30.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:30.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:30.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:30.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:30.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:30.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:30.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:30.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:30.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:30.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:31.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4467244 2021-05-12T12:47:31.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4467244 '!=' null 2021-05-12T12:47:31.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:31.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:31.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:31.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:31.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:31.407 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:31.407 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:31.407 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:31.407 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:31.407 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:31.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:31.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:31.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:31.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:31.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:32.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:32.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:32.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:32.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:32.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:32.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:32.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:32.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:32.356 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:32.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621008 2021-05-12T12:47:32.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621008 '!=' null 2021-05-12T12:47:32.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:32.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:32.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:32.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:32.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:32.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:32.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:32.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:32.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:32.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:32.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:33.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:33.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:33.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:33.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:33.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:33.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:33.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:33.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:33.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:33.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:33.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:33.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:33.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:34.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494656 2021-05-12T12:47:34.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494656 '!=' null 2021-05-12T12:47:34.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:34.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:34.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:34.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:34.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:34.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:34.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:34.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:34.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:34.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:34.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:34.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:34.758 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:34.758 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:34.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:35.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:35.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:35.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:35.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:35.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:35.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:35.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:35.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:35.243 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:35.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620976 2021-05-12T12:47:35.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620976 '!=' null 2021-05-12T12:47:35.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:35.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:35.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:35.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:35.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:35.835 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:35.835 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:35.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:35.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:35.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:35.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:36.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:36.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:36.195 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:36.195 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:36.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:36.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:36.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:36.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:36.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:36.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:36.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:36.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:36.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:37.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620976 2021-05-12T12:47:37.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620976 '!=' null 2021-05-12T12:47:37.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:37.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:37.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:37.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:37.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:37.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:37.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:37.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:37.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:37.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:37.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:37.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:37.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:37.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:37.625 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:38.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:38.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:38.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:38.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:38.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:38.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:38.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:38.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:38.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:38.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620957 2021-05-12T12:47:38.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620957 '!=' null 2021-05-12T12:47:38.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:38.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:38.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:38.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:38.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:38.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:38.701 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:38.702 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:38.702 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:38.702 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:38.702 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:39.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:39.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:39.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:39.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:39.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:39.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:39.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:39.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:39.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:39.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:39.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:39.549 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:39.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:40.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368269 2021-05-12T12:47:40.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368269 '!=' null 2021-05-12T12:47:40.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:40.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:40.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:40.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:40.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:40.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:40.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:40.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:40.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:40.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:40.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:40.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:40.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:40.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:40.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:40.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:40.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:40.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:40.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:40.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:40.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:40.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:40.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:40.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:41.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620944 2021-05-12T12:47:41.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620944 '!=' null 2021-05-12T12:47:41.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:41.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:41.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:41.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:41.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:41.569 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:41.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:41.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:41.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:41.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:41.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:41.943 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:41.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:41.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:41.944 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:42.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:42.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:42.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:42.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:42.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:42.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:42.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:42.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:42.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:42.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620944 2021-05-12T12:47:42.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620944 '!=' null 2021-05-12T12:47:42.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:42.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:42.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:43.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:43.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:43.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:43.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:43.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:43.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:43.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:43.014 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:43.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:43.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:43.384 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:43.384 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:43.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:43.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:43.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:43.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:43.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:43.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:43.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:43.957 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:43.957 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:44.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368279 2021-05-12T12:47:44.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368279 '!=' null 2021-05-12T12:47:44.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:44.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:44.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:44.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:44.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:44.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:44.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:44.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:44.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:44.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:44.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:44.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:44.901 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:44.902 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:44.902 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:45.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:45.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:45.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:45.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:45.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:45.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:45.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:45.392 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:45.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:45.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494636 2021-05-12T12:47:45.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494636 '!=' null 2021-05-12T12:47:45.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:45.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:45.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:46.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:46.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:46.006 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:46.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:46.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:46.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:46.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:46.007 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:46.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:46.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:46.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:46.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:46.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:46.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:46.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:46.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:46.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:46.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:46.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:46.926 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:46.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:47.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494648 2021-05-12T12:47:47.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494648 '!=' null 2021-05-12T12:47:47.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:47.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:47.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:47.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:47.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:47.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:47.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:47.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:47.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:47.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:47.531 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:47.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:47.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:47.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:47.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:48.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:48.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:48.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:48.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:48.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:48.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:48.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:48.368 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:48.368 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:48.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494648 2021-05-12T12:47:48.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494648 '!=' null 2021-05-12T12:47:48.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:48.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:48.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:48.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:48.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:48.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:48.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:48.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:48.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:48.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:48.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:49.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:49.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:49.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:49.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:49.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:49.812 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:49.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:49.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:49.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:49.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:49.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:49.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:49.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:50.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368319 2021-05-12T12:47:50.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368319 '!=' null 2021-05-12T12:47:50.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:50.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:50.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:50.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:50.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:50.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:50.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:50.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:50.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:50.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:50.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:50.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:50.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:50.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:50.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:51.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:51.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:51.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:51.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:51.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:51.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:51.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:51.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:51.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:51.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620963 2021-05-12T12:47:51.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620963 '!=' null 2021-05-12T12:47:51.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:51.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:51.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:51.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:51.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:51.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:51.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:51.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:51.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:51.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:51.834 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:52.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:52.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:52.192 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:52.192 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:52.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:52.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:52.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:52.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:52.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:52.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:52.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:52.683 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:52.683 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:53.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620963 2021-05-12T12:47:53.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620963 '!=' null 2021-05-12T12:47:53.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:53.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:53.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:53.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:53.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:53.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:53.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:53.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:53.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:53.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:53.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:53.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:53.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:53.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:53.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:54.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:54.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:54.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:54.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:54.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:54.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:54.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:54.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:54.117 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:54.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368266 2021-05-12T12:47:54.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368266 '!=' null 2021-05-12T12:47:54.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:54.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:54.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:54.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:54.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:54.708 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:54.708 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:54.708 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:54.709 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:54.709 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:54.709 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:55.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:55.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:55.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:55.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:55.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:55.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:55.556 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:55.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:55.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:55.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:55.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:55.557 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:55.558 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:56.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620901 2021-05-12T12:47:56.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620901 '!=' null 2021-05-12T12:47:56.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:56.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:56.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:56.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:56.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:56.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:56.147 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:56.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:56.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:56.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:56.148 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:56.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:56.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:56.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:56.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:56.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:56.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:56.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:56.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:56.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:56.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:56.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:56.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:56.974 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:57.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620902 2021-05-12T12:47:57.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620902 '!=' null 2021-05-12T12:47:57.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:57.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:57.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:57.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:57.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:57.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:57.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:57.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:57.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:57.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:57.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:57.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:57.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:57.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:57.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:58.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:58.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:58.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:58.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:58.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:58.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:58.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:58.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:58.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:58.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620902 2021-05-12T12:47:58.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620902 '!=' null 2021-05-12T12:47:58.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:58.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:58.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:58.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:59.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:59.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:59.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:59.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:59.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:59.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:59.002 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:59.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:59.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:59.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:59.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:59.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:59.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:59.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:59.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:59.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:59.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:59.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:59.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:59.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:00.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494522 2021-05-12T12:48:00.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494522 '!=' null 2021-05-12T12:48:00.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:00.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:00.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:00.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:00.416 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:00.417 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:00.417 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:00.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:00.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:00.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:00.418 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:00.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:00.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:00.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:00.777 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:01.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:01.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:01.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:01.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:01.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:01.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:01.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:01.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:01.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:01.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620909 2021-05-12T12:48:01.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620909 '!=' null 2021-05-12T12:48:01.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:01.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:01.765 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:01.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:01.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:01.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:01.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:01.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:01.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:01.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:01.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:02.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:02.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:02.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:02.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:02.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:02.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:02.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:02.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:02.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:02.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:02.721 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:02.722 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:02.722 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:03.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620909 2021-05-12T12:48:03.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620909 '!=' null 2021-05-12T12:48:03.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:03.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:03.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:03.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:03.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:03.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:03.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:03.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:03.313 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:03.313 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:03.313 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:03.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:03.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:03.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:03.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:04.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:04.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:04.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:04.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:04.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:04.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:04.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:04.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:04.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:04.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368233 2021-05-12T12:48:04.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368233 '!=' null 2021-05-12T12:48:04.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:04.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:04.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:04.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:04.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:04.736 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:04.736 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:04.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:04.737 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:04.738 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:04.738 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:05.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:05.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:05.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:05.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:05.566 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:05.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:05.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:05.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:05.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:05.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:05.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:05.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:05.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:06.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620933 2021-05-12T12:48:06.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620933 '!=' null 2021-05-12T12:48:06.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:06.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:06.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:06.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:06.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:06.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:06.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:06.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:06.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:06.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:06.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:06.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:06.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:06.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:06.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:07.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:07.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:07.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:07.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:07.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:07.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:07.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:07.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:07.027 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:07.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620997 2021-05-12T12:48:07.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620997 '!=' null 2021-05-12T12:48:07.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:07.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:07.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:07.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:07.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:07.628 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:07.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:07.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:07.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:07.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:07.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:07.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:07.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:07.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:07.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:08.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:08.486 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:08.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:08.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:08.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:08.487 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:08.488 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:08.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:08.488 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:08.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620997 2021-05-12T12:48:08.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620997 '!=' null 2021-05-12T12:48:08.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:08.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:08.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:09.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:09.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:09.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:09.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:09.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:09.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:09.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:09.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:09.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:09.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:09.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:09.443 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:09.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:09.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:09.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:09.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:09.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:09.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:09.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:09.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:09.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:10.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368341 2021-05-12T12:48:10.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368341 '!=' null 2021-05-12T12:48:10.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:10.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:10.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:10.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:10.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:10.522 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:10.522 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:10.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:10.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:10.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:10.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:10.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:10.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:10.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:10.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:11.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:11.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:11.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:11.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:11.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:11.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:11.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:11.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:11.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:11.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494681 2021-05-12T12:48:11.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494681 '!=' null 2021-05-12T12:48:11.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:11.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:11.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:12.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:12.017 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:12.018 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:12.019 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:12.019 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:12.019 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:12.019 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:12.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:12.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:12.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:12.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:12.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:12.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:12.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:12.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:12.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:12.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:12.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:12.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:12.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:12.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:13.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368355 2021-05-12T12:48:13.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368355 '!=' null 2021-05-12T12:48:13.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:13.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:13.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:13.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:13.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:13.457 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:13.457 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:13.457 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:13.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:13.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:13.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:13.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:13.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:13.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:13.811 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:14.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:14.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:14.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:14.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:14.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:14.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:14.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:14.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:14.297 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:14.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368355 2021-05-12T12:48:14.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368355 '!=' null 2021-05-12T12:48:14.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:14.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:14.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:14.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:14.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:14.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:14.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:14.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:14.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:14.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:14.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:15.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:15.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:15.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:15.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:15.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:15.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:15.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:15.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:15.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:15.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:15.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:15.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:15.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:16.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621009 2021-05-12T12:48:16.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621009 '!=' null 2021-05-12T12:48:16.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:16.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:16.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:16.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:16.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:16.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:16.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:16.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:16.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:16.311 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:16.311 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:16.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:16.666 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:16.667 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:16.667 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:17.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:17.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:17.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:17.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:17.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:17.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:17.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:17.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:17.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:17.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621036 2021-05-12T12:48:17.657 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621036 '!=' null 2021-05-12T12:48:17.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:17.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:17.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:17.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:17.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:17.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:17.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:17.759 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:17.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:17.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:17.760 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:18.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:18.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:18.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:18.123 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:18.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:18.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:18.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:18.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:18.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:18.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:18.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:18.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:18.617 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:19.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621036 2021-05-12T12:48:19.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621036 '!=' null 2021-05-12T12:48:19.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:19.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:19.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:19.195 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:19.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:19.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:19.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:19.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:19.197 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:19.198 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:19.198 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:19.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:19.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:19.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:19.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:20.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:20.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:20.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:20.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:20.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:20.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:20.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:20.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:20.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:20.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368388 2021-05-12T12:48:20.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368388 '!=' null 2021-05-12T12:48:20.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:20.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:20.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:20.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:20.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:20.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:20.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:20.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:20.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:20.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:20.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:21.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:21.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:21.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:21.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:21.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:21.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:21.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:21.493 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:21.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:21.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:21.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:21.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:21.496 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:22.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621020 2021-05-12T12:48:22.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621020 '!=' null 2021-05-12T12:48:22.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:22.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:22.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:22.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:22.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:22.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:22.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:22.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:22.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:22.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:22.105 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:22.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:22.464 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:22.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:22.465 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:22.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:22.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:22.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:22.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:22.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:22.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:22.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:22.949 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:22.950 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:23.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494684 2021-05-12T12:48:23.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494684 '!=' null 2021-05-12T12:48:23.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:23.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:23.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:23.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:23.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:23.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:23.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:23.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:23.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:23.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:23.551 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:23.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:23.910 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:23.910 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:23.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:24.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:24.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:24.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:24.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:24.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:24.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:24.401 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:24.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:24.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:24.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494684 2021-05-12T12:48:24.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494684 '!=' null 2021-05-12T12:48:24.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:24.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:24.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:25.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:25.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:25.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:25.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:25.003 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:25.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:25.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:25.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:25.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:25.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:25.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:25.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:25.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:25.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:25.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:25.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:25.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:25.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:25.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:25.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:25.838 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:26.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621011 2021-05-12T12:48:26.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621011 '!=' null 2021-05-12T12:48:26.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:26.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:26.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:26.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:26.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:26.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:26.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:26.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:26.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:26.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:26.425 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:26.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:26.782 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:26.783 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:26.783 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:27.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:27.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:27.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:27.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:27.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:27.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:27.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:27.267 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:27.267 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:27.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621010 2021-05-12T12:48:27.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621010 '!=' null 2021-05-12T12:48:27.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:27.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:27.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:27.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:27.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:27.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:27.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:27.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:27.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:27.939 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:27.940 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:28.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:28.378 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:28.378 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:28.378 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:28.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:28.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:28.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:28.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:28.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:28.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:28.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:28.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:28.877 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:29.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368312 2021-05-12T12:48:29.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368312 '!=' null 2021-05-12T12:48:29.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:29.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:29.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:29.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:29.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:29.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:29.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:29.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:29.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:29.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:29.461 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:29.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:29.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:29.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:29.827 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:30.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:30.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:30.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:30.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:30.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:30.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:30.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:30.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:30.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:30.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368312 2021-05-12T12:48:30.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368312 '!=' null 2021-05-12T12:48:30.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:30.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:30.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:30.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:30.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:30.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:30.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:30.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:30.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:30.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:30.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:31.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:31.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:31.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:31.251 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:31.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:31.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:31.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:31.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:31.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:31.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:31.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:31.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:31.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:32.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620986 2021-05-12T12:48:32.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620986 '!=' null 2021-05-12T12:48:32.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:32.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:32.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:32.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:32.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:32.305 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:32.305 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:32.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:32.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:32.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:32.306 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:32.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:32.670 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:32.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:32.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:33.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:33.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:33.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:33.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:33.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:33.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:33.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:33.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:33.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:33.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368316 2021-05-12T12:48:33.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368316 '!=' null 2021-05-12T12:48:33.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:33.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:33.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:33.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:33.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:33.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:33.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:33.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:33.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:33.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:33.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:34.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:34.106 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:34.106 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:34.106 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:34.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:34.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:34.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:34.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:34.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:34.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:34.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:34.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:34.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:35.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368316 2021-05-12T12:48:35.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368316 '!=' null 2021-05-12T12:48:35.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:35.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:35.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:35.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:35.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:35.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:35.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:35.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:35.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:35.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:35.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:35.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:35.561 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:35.561 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:35.561 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:36.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:36.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:36.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:36.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:36.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:36.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:36.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:36.053 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:36.053 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:36.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494645 2021-05-12T12:48:36.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494645 '!=' null 2021-05-12T12:48:36.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:36.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:36.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:36.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:36.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:36.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:36.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:36.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:36.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:36.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:36.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:36.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:36.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:36.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:36.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:37.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:37.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:37.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:37.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:37.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:37.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:37.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:37.477 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:37.477 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:37.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494628 2021-05-12T12:48:37.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494628 '!=' null 2021-05-12T12:48:37.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:37.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:37.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:38.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:38.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:38.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:38.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:38.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:38.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:38.075 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:38.076 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:38.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:38.428 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:38.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:38.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:38.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:38.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:38.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:38.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:38.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:38.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:38.915 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:38.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:38.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:39.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368253 2021-05-12T12:48:39.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368253 '!=' null 2021-05-12T12:48:39.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:39.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:39.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:39.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:39.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:39.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:39.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:39.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:39.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:39.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:39.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:39.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:39.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:39.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:39.971 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:40.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:40.450 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:40.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:40.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:40.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:40.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:40.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:40.452 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:40.453 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:40.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368253 2021-05-12T12:48:40.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368253 '!=' null 2021-05-12T12:48:40.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:40.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:40.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:41.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:41.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:41.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:41.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:41.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:41.078 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:41.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:41.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:41.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:41.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:41.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:41.430 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:41.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:41.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:41.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:41.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:41.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:41.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:41.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:41.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:41.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:42.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=1310470 2021-05-12T12:48:42.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 1310470 '!=' null 2021-05-12T12:48:42.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:42.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:42.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:42.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:42.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:42.516 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:42.517 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:42.517 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:42.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:42.518 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:42.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:42.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:42.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:42.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:42.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:43.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:43.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:43.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:43.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:43.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:43.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:43.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:43.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:43.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:43.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620923 2021-05-12T12:48:43.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620923 '!=' null 2021-05-12T12:48:43.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:43.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:43.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:43.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:43.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:43.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:43.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:43.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:43.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:43.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:43.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:44.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T12:48:44.283 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:44.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:44.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:44.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T12:48:44.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:48:44.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:48:44.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:522: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2021-05-12T12:48:44.751 INFO:tasks.workunit.client.0.smithi191.stdout:TEST PASSED 2021-05-12T12:48:44.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:526: TEST_recovery_pool_priority: echo TEST PASSED 2021-05-12T12:48:44.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: delete_pool test1 2021-05-12T12:48:44.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2021-05-12T12:48:44.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-05-12T12:48:45.229 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test1' does not exist 2021-05-12T12:48:45.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:530: TEST_recovery_pool_priority: delete_pool test2 2021-05-12T12:48:45.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2021-05-12T12:48:45.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-05-12T12:48:45.957 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test2' does not exist 2021-05-12T12:48:45.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:531: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2021-05-12T12:48:45.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:48:45.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:48:45.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:48:45.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:48:45.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:48:46.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:48:46.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: return 0 2021-05-12T12:48:46.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: teardown td/osd-recovery-prio 2021-05-12T12:48:46.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-05-12T12:48:46.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:48:46.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-05-12T12:48:46.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:48:46.077 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:48:46.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:48:46.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:48:46.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:48:46.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:48:46.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:48:46.080 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:48:46.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:48:46.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:48:46.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:48:46.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:48:46.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:48:46.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:48:46.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:48:46.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:48:46.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:48:46.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:48:46.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-05-12T12:48:46.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:48:46.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:48:46.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:48:46.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.190757 2021-05-12T12:48:46.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:48:46.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:48:46.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2021-05-12T12:48:46.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2021-05-12T12:48:46.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2021-05-12T12:48:46.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2021-05-12T12:48:46.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-05-12T12:48:46.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:48:46.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-05-12T12:48:46.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:48:46.893 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:48:46.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:48:46.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:48:46.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:48:46.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:48:46.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:48:46.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:48:46.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:48:46.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:48:46.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:48:46.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:48:46.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:48:46.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:48:46.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:48:46.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:48:46.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:48:46.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:48:46.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-05-12T12:48:46.900 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:48:46.900 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:48:46.900 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:48:46.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.190757 2021-05-12T12:48:46.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:48:46.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:48:46.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2021-05-12T12:48:46.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:48:46.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:48:46.904 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:48:46.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.190757 2021-05-12T12:48:46.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:48:46.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:48:46.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:48:46.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2021-05-12T12:48:46.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_priority td/osd-recovery-prio 2021-05-12T12:48:46.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:44: TEST_recovery_priority: local dir=td/osd-recovery-prio 2021-05-12T12:48:46.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:45: TEST_recovery_priority: local pools=10 2021-05-12T12:48:46.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:46: TEST_recovery_priority: local OSDS=5 2021-05-12T12:48:46.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local max_tries=10 2021-05-12T12:48:46.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2021-05-12T12:48:46.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2021-05-12T12:48:46.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:48:46.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:48:46.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:48:46.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2021-05-12T12:48:46.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2021-05-12T12:48:47.267 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:48:47.267 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:48:47.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:48:47.268 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:48:47.268 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:48:47.268 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:48:47.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:48:47.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:48:47.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:48:47.512 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:48:47.512 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:48:47.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:48:47.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:48:47.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:48:47.513 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:48:47.514 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:48:47.514 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:48:47.514 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:48:47.514 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:48:47.515 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:48:47.515 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-mon.a.asok 2021-05-12T12:48:47.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:48:47.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.190757/ceph-mon.a.asok config get fsid 2021-05-12T12:48:47.608 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:48:47.609 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:48:47.609 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:48:47.609 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:48:47.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:48:47.610 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:48:47.610 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:48:47.610 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:48:47.611 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:48:47.611 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:48:47.611 INFO:tasks.workunit.client.0.smithi191.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:48:47.611 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-mon.a.asok 2021-05-12T12:48:47.611 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:48:47.612 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.190757/ceph-mon.a.asok config get mon_host 2021-05-12T12:48:47.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2021-05-12T12:48:47.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2021-05-12T12:48:47.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:48:47.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:48:47.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:48:47.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2021-05-12T12:48:47.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:48:48.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:48:48.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:48:48.566 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:48:48.566 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:48:48.566 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:48:48.566 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:48:48.567 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:48:48.567 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:48:48.567 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:48:48.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:51: TEST_recovery_priority: export CEPH_ARGS 2021-05-12T12:48:48.589 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: expr 5 - 1 2021-05-12T12:48:48.590 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: seq 0 4 2021-05-12T12:48:48.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:48:48.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2021-05-12T12:48:48.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:48:48.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:48:48.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:48:48.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:48:48.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2021-05-12T12:48:48.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:48:48.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:48:48.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:48:48.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:48:48.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2021-05-12T12:48:48.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2021-05-12T12:48:48.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:48:48.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:48:48.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:48:48.598 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:48:48.598 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:48:48.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:48:48.599 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:48:48.599 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:48:48.599 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:48:48.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:48:48.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:48:48.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:48:48.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:48:48.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:48:48.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:48:48.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:48:48.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:48:48.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:48:48.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:48:48.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:48:48.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:48:48.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/0 2021-05-12T12:48:48.602 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:48:48.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e86f24ec-81ac-4391-98f6-cce463c30ec4 2021-05-12T12:48:48.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 e86f24ec-81ac-4391-98f6-cce463c30ec4' 2021-05-12T12:48:48.603 INFO:tasks.workunit.client.0.smithi191.stdout:add osd0 e86f24ec-81ac-4391-98f6-cce463c30ec4 2021-05-12T12:48:48.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:48:48.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCwzptg9ocdJBAAiqvFb0PUksqyJrB9NJh6jg== 2021-05-12T12:48:48.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCwzptg9ocdJBAAiqvFb0PUksqyJrB9NJh6jg=="}' 2021-05-12T12:48:48.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e86f24ec-81ac-4391-98f6-cce463c30ec4 -i td/osd-recovery-prio/0/new.json 2021-05-12T12:48:48.919 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:48:48.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/0/new.json 2021-05-12T12:48:48.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCwzptg9ocdJBAAiqvFb0PUksqyJrB9NJh6jg== --osd-uuid e86f24ec-81ac-4391-98f6-cce463c30ec4 2021-05-12T12:48:48.954 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:48.950+0000 7fc12e63f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:48:48.955 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:48.952+0000 7fc12e63f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:48:48.957 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:48.953+0000 7fc12e63f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:48:48.958 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:48.954+0000 7fc12e63f0c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:48:48.958 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:48.954+0000 7fc12e63f0c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:48:48.958 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:48.954+0000 7fc12e63f0c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:48:49.003 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:48.999+0000 7fc12e63f0c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2021-05-12T12:48:53.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2021-05-12T12:48:53.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:48:53.631 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:48:53.632 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd0 key to auth repository 2021-05-12T12:48:53.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:48:54.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:48:54.109 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.0 2021-05-12T12:48:54.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:48:54.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:48:54.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:48:54.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:48:54.130 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:54.126+0000 7fb4a2eb00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:48:54.132 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:54.128+0000 7fb4a2eb00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:48:54.133 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:54.129+0000 7fb4a2eb00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:48:54.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:48:54.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:48:54.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:48:54.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:48:54.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:48:54.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:48:54.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:48:54.476 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:48:54.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:48:54.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:48:54.649 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:54.645+0000 7fb4a2eb00c0 -1 Falling back to public interface 2021-05-12T12:48:54.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:48:55.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:48:55.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:48:55.843 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:48:55.846 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:48:55.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:48:55.847 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:48:56.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:48:56.859 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:56.856+0000 7fb4a2eb00c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:48:57.203 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:48:57.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:48:57.204 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:48:57.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:48:57.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:48:57.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:48:57.575 INFO:tasks.workunit.client.0.smithi191.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/247774,v1:127.0.0.1:6803/247774] [v2:127.0.0.1:6804/247774,v1:127.0.0.1:6805/247774] exists,up e86f24ec-81ac-4391-98f6-cce463c30ec4 2021-05-12T12:48:57.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:48:57.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:48:57.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:48:57.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:48:57.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2021-05-12T12:48:57.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:48:57.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:48:57.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:48:57.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:48:57.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2021-05-12T12:48:57.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:48:57.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:48:57.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:48:57.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:48:57.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2021-05-12T12:48:57.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2021-05-12T12:48:57.584 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:48:57.584 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:48:57.584 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:48:57.584 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:48:57.584 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:48:57.585 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:48:57.585 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:48:57.585 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:48:57.585 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:48:57.585 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:48:57.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:48:57.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:48:57.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:48:57.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:48:57.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:48:57.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:48:57.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:48:57.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:48:57.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:48:57.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:48:57.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:48:57.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/1 2021-05-12T12:48:57.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:48:57.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=5dc275d5-0327-4363-af1e-9e851b0877f4 2021-05-12T12:48:57.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 5dc275d5-0327-4363-af1e-9e851b0877f4' 2021-05-12T12:48:57.589 INFO:tasks.workunit.client.0.smithi191.stdout:add osd1 5dc275d5-0327-4363-af1e-9e851b0877f4 2021-05-12T12:48:57.590 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:48:57.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC5zptgzOYuIxAAKB2LCwexdBF6esIu70Hd1g== 2021-05-12T12:48:57.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC5zptgzOYuIxAAKB2LCwexdBF6esIu70Hd1g=="}' 2021-05-12T12:48:57.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 5dc275d5-0327-4363-af1e-9e851b0877f4 -i td/osd-recovery-prio/1/new.json 2021-05-12T12:48:58.047 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:48:58.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/1/new.json 2021-05-12T12:48:58.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC5zptgzOYuIxAAKB2LCwexdBF6esIu70Hd1g== --osd-uuid 5dc275d5-0327-4363-af1e-9e851b0877f4 2021-05-12T12:48:58.082 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:58.078+0000 7f4d8b2c30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:48:58.084 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:58.080+0000 7f4d8b2c30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:48:58.085 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:58.082+0000 7f4d8b2c30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:48:58.086 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:58.082+0000 7f4d8b2c30c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:48:58.086 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:58.083+0000 7f4d8b2c30c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:48:58.087 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:58.083+0000 7f4d8b2c30c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:48:58.132 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:48:58.129+0000 7f4d8b2c30c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2021-05-12T12:49:02.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2021-05-12T12:49:02.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:49:02.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:49:02.901 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd1 key to auth repository 2021-05-12T12:49:02.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:49:03.398 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.1 2021-05-12T12:49:03.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:49:03.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:03.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:49:03.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:49:03.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:49:03.420 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:03.417+0000 7ffa627b50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:03.422 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:03.418+0000 7ffa627b50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:03.423 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:03.420+0000 7ffa627b50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:03.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:49:03.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:49:03.777 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:49:03.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:49:03.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:49:03.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:03.779 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:49:03.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:49:03.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:03.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:49:03.939 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:03.935+0000 7ffa627b50c0 -1 Falling back to public interface 2021-05-12T12:49:04.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:05.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:05.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:05.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:49:05.138 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:49:05.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:05.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:49:05.508 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:06.159 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:06.155+0000 7ffa627b50c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:49:06.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:06.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:06.510 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:49:06.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:49:06.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:06.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:49:06.920 INFO:tasks.workunit.client.0.smithi191.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/249159,v1:127.0.0.1:6811/249159] [v2:127.0.0.1:6812/249159,v1:127.0.0.1:6813/249159] exists,up 5dc275d5-0327-4363-af1e-9e851b0877f4 2021-05-12T12:49:06.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:49:06.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:49:06.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:49:06.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:49:06.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2021-05-12T12:49:06.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:49:06.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:49:06.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:49:06.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:49:06.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2021-05-12T12:49:06.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:49:06.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:49:06.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:49:06.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:49:06.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2021-05-12T12:49:06.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2021-05-12T12:49:06.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:49:06.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:49:06.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:49:06.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:49:06.929 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:06.930 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:06.930 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:06.930 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:06.930 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:49:06.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:49:06.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:49:06.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:49:06.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:49:06.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:49:06.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:49:06.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:49:06.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:49:06.932 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:49:06.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:06.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:49:06.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:49:06.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/2 2021-05-12T12:49:06.934 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:49:06.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=41ed4ff5-7c43-4317-a78c-f2960dffbf73 2021-05-12T12:49:06.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 41ed4ff5-7c43-4317-a78c-f2960dffbf73' 2021-05-12T12:49:06.934 INFO:tasks.workunit.client.0.smithi191.stdout:add osd2 41ed4ff5-7c43-4317-a78c-f2960dffbf73 2021-05-12T12:49:06.935 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:49:06.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDCzptgHMO3NxAAQl4m1atnaldc1eQIhGWTqQ== 2021-05-12T12:49:06.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDCzptgHMO3NxAAQl4m1atnaldc1eQIhGWTqQ=="}' 2021-05-12T12:49:06.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 41ed4ff5-7c43-4317-a78c-f2960dffbf73 -i td/osd-recovery-prio/2/new.json 2021-05-12T12:49:07.557 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:49:07.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/2/new.json 2021-05-12T12:49:07.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDCzptgHMO3NxAAQl4m1atnaldc1eQIhGWTqQ== --osd-uuid 41ed4ff5-7c43-4317-a78c-f2960dffbf73 2021-05-12T12:49:07.593 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:07.589+0000 7f7dbe3920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:07.609 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:07.605+0000 7f7dbe3920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:07.610 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:07.607+0000 7f7dbe3920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:07.611 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:07.608+0000 7f7dbe3920c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-05-12T12:49:07.612 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:07.608+0000 7f7dbe3920c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-05-12T12:49:07.612 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:07.608+0000 7f7dbe3920c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-05-12T12:49:07.682 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:07.679+0000 7f7dbe3920c0 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2021-05-12T12:49:12.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2021-05-12T12:49:12.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:49:12.250 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd2 key to auth repository 2021-05-12T12:49:12.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:49:12.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:49:12.735 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.2 2021-05-12T12:49:12.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:49:12.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:12.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:49:12.736 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:49:12.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:49:12.756 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:12.753+0000 7f0cc19230c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:12.758 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:12.755+0000 7f0cc19230c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:12.760 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:12.756+0000 7f0cc19230c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:13.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:49:13.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:49:13.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:49:13.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:49:13.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:49:13.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:13.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:49:13.112 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:49:13.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:13.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:49:13.285 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:13.281+0000 7f0cc19230c0 -1 Falling back to public interface 2021-05-12T12:49:13.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:14.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:14.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:14.502 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:49:14.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:49:14.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:14.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:49:14.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:15.528 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:15.525+0000 7f0cc19230c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:49:15.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:15.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:15.868 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:49:15.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:49:15.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:15.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:49:16.260 INFO:tasks.workunit.client.0.smithi191.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/250544,v1:127.0.0.1:6819/250544] [v2:127.0.0.1:6820/250544,v1:127.0.0.1:6821/250544] exists,up 41ed4ff5-7c43-4317-a78c-f2960dffbf73 2021-05-12T12:49:16.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:49:16.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:49:16.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:49:16.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:49:16.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2021-05-12T12:49:16.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:49:16.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:49:16.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-05-12T12:49:16.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:49:16.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2021-05-12T12:49:16.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:49:16.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:49:16.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:49:16.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:49:16.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2021-05-12T12:49:16.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2021-05-12T12:49:16.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:49:16.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:49:16.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:49:16.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:49:16.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:16.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:16.270 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:16.270 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:16.270 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:49:16.271 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:49:16.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:49:16.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:49:16.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:49:16.271 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:49:16.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:49:16.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:49:16.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:49:16.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:49:16.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:16.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:49:16.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:49:16.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/3 2021-05-12T12:49:16.273 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:49:16.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f81ca3f9-c32d-4577-be96-0293a3c2582a 2021-05-12T12:49:16.274 INFO:tasks.workunit.client.0.smithi191.stdout:add osd3 f81ca3f9-c32d-4577-be96-0293a3c2582a 2021-05-12T12:49:16.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 f81ca3f9-c32d-4577-be96-0293a3c2582a' 2021-05-12T12:49:16.275 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:49:16.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDMzptgD8xoEBAA7MR3BGOIlerRrMtWRzwFrA== 2021-05-12T12:49:16.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDMzptgD8xoEBAA7MR3BGOIlerRrMtWRzwFrA=="}' 2021-05-12T12:49:16.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f81ca3f9-c32d-4577-be96-0293a3c2582a -i td/osd-recovery-prio/3/new.json 2021-05-12T12:49:16.824 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:49:16.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/3/new.json 2021-05-12T12:49:16.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDMzptgD8xoEBAA7MR3BGOIlerRrMtWRzwFrA== --osd-uuid f81ca3f9-c32d-4577-be96-0293a3c2582a 2021-05-12T12:49:16.859 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:16.856+0000 7f33f38430c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:16.861 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:16.858+0000 7f33f38430c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:16.862 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:16.859+0000 7f33f38430c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:16.864 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:16.860+0000 7f33f38430c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-05-12T12:49:16.864 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:16.860+0000 7f33f38430c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-05-12T12:49:16.864 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:16.860+0000 7f33f38430c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-05-12T12:49:16.901 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:16.898+0000 7f33f38430c0 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2021-05-12T12:49:21.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2021-05-12T12:49:21.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:49:21.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-05-12T12:49:21.731 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd3 key to auth repository 2021-05-12T12:49:21.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:49:22.222 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.3 2021-05-12T12:49:22.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-05-12T12:49:22.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:22.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:49:22.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:49:22.224 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:49:22.243 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:22.240+0000 7f16d210b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:22.245 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:22.242+0000 7f16d210b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:22.246 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:22.243+0000 7f16d210b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:22.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-05-12T12:49:22.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:49:22.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:49:22.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:49:22.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:49:22.584 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:22.584 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:49:22.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:49:22.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:22.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:49:22.764 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:22.761+0000 7f16d210b0c0 -1 Falling back to public interface 2021-05-12T12:49:22.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:23.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:23.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:23.966 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:49:23.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:49:23.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:23.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:49:24.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:25.056 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:25.053+0000 7f16d210b0c0 -1 osd.3 0 log_to_monitors {default=true} 2021-05-12T12:49:25.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:25.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:25.332 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:49:25.332 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:49:25.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:25.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:49:25.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:26.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:26.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:26.748 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:49:26.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:49:26.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:26.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:49:27.109 INFO:tasks.workunit.client.0.smithi191.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/251929,v1:127.0.0.1:6827/251929] [v2:127.0.0.1:6828/251929,v1:127.0.0.1:6829/251929] exists,up f81ca3f9-c32d-4577-be96-0293a3c2582a 2021-05-12T12:49:27.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:49:27.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:49:27.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:49:27.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:49:27.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2021-05-12T12:49:27.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:49:27.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:49:27.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-05-12T12:49:27.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:49:27.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2021-05-12T12:49:27.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:49:27.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:49:27.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:49:27.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:49:27.119 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2021-05-12T12:49:27.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2021-05-12T12:49:27.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:49:27.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:49:27.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:49:27.121 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:49:27.121 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:27.121 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:27.121 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:27.121 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:27.122 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:49:27.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:49:27.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' 2021-05-12T12:49:27.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:49:27.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:49:27.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:49:27.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:49:27.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:49:27.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:49:27.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:49:27.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:27.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:49:27.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:49:27.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/4 2021-05-12T12:49:27.125 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:49:27.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=5ab2d041-37f3-4ebd-931a-0ce9dcf9514d 2021-05-12T12:49:27.126 INFO:tasks.workunit.client.0.smithi191.stdout:add osd4 5ab2d041-37f3-4ebd-931a-0ce9dcf9514d 2021-05-12T12:49:27.126 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd4 5ab2d041-37f3-4ebd-931a-0ce9dcf9514d' 2021-05-12T12:49:27.126 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:49:27.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDXzptglmpjBxAAMqAuszIzZ5Te9+6zjG8Fug== 2021-05-12T12:49:27.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDXzptglmpjBxAAMqAuszIzZ5Te9+6zjG8Fug=="}' 2021-05-12T12:49:27.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 5ab2d041-37f3-4ebd-931a-0ce9dcf9514d -i td/osd-recovery-prio/4/new.json 2021-05-12T12:49:27.588 INFO:tasks.workunit.client.0.smithi191.stdout:4 2021-05-12T12:49:27.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/4/new.json 2021-05-12T12:49:27.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 4 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDXzptglmpjBxAAMqAuszIzZ5Te9+6zjG8Fug== --osd-uuid 5ab2d041-37f3-4ebd-931a-0ce9dcf9514d 2021-05-12T12:49:27.623 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:27.619+0000 7fedcbaac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:27.624 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:27.621+0000 7fedcbaac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:27.626 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:27.623+0000 7fedcbaac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:27.627 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:27.623+0000 7fedcbaac0c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-05-12T12:49:27.627 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:27.623+0000 7fedcbaac0c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-05-12T12:49:27.627 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:27.623+0000 7fedcbaac0c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-05-12T12:49:27.708 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:27.704+0000 7fedcbaac0c0 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2021-05-12T12:49:32.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2021-05-12T12:49:32.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:49:32.666 INFO:tasks.workunit.client.0.smithi191.stdout:adding osd4 key to auth repository 2021-05-12T12:49:32.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd4 key to auth repository 2021-05-12T12:49:32.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:49:33.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.4 2021-05-12T12:49:33.151 INFO:tasks.workunit.client.0.smithi191.stdout:start osd.4 2021-05-12T12:49:33.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:49:33.152 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 4 --fsid=e65e3fc9-ba98-4f25-861e-5f0c7c30d212 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.190757/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:33.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:49:33.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:49:33.173 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:33.169+0000 7fb41006e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:33.175 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:33.171+0000 7fb41006e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:33.175 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:33.172+0000 7fb41006e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:33.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 4 2021-05-12T12:49:33.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:49:33.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=4 2021-05-12T12:49:33.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:49:33.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:49:33.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:33.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:49:33.531 INFO:tasks.workunit.client.0.smithi191.stdout:0 2021-05-12T12:49:33.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:33.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.4 up' 2021-05-12T12:49:33.691 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:33.688+0000 7fb41006e0c0 -1 Falling back to public interface 2021-05-12T12:49:33.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:34.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:34.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:34.889 INFO:tasks.workunit.client.0.smithi191.stdout:1 2021-05-12T12:49:34.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:49:34.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:34.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.4 up' 2021-05-12T12:49:35.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:35.944 INFO:tasks.workunit.client.0.smithi191.stderr:2021-05-12T12:49:35.941+0000 7fb41006e0c0 -1 osd.4 0 log_to_monitors {default=true} 2021-05-12T12:49:36.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:36.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:36.255 INFO:tasks.workunit.client.0.smithi191.stdout:2 2021-05-12T12:49:36.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:49:36.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:36.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.4 up' 2021-05-12T12:49:36.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:37.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:37.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:37.677 INFO:tasks.workunit.client.0.smithi191.stdout:3 2021-05-12T12:49:37.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:49:37.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:37.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.4 up' 2021-05-12T12:49:38.047 INFO:tasks.workunit.client.0.smithi191.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/253348,v1:127.0.0.1:6835/253348] [v2:127.0.0.1:6836/253348,v1:127.0.0.1:6837/253348] exists,up 5ab2d041-37f3-4ebd-931a-0ce9dcf9514d 2021-05-12T12:49:38.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:49:38.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:49:38.049 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:49:38.049 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: seq 1 10 2021-05-12T12:49:38.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:49:38.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test1 1 1 2021-05-12T12:49:38.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-05-12T12:49:38.578 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test1' already exists 2021-05-12T12:49:38.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:49:39.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test1 size 2 2021-05-12T12:49:40.563 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 1 size to 2 2021-05-12T12:49:40.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:49:40.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test2 1 1 2021-05-12T12:49:40.573 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2021-05-12T12:49:41.081 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test2' already exists 2021-05-12T12:49:41.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:49:42.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test2 size 2 2021-05-12T12:49:42.810 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 2 size to 2 2021-05-12T12:49:42.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:49:42.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test3 1 1 2021-05-12T12:49:42.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2021-05-12T12:49:43.354 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test3' already exists 2021-05-12T12:49:43.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:49:44.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test3 size 2 2021-05-12T12:49:45.177 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 3 size to 2 2021-05-12T12:49:45.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:49:45.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test4 1 1 2021-05-12T12:49:45.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2021-05-12T12:49:45.817 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test4' already exists 2021-05-12T12:49:45.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:49:46.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test4 size 2 2021-05-12T12:49:47.657 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 4 size to 2 2021-05-12T12:49:47.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:49:47.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test5 1 1 2021-05-12T12:49:47.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2021-05-12T12:49:48.212 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test5' already exists 2021-05-12T12:49:48.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:49:49.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test5 size 2 2021-05-12T12:49:50.112 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 5 size to 2 2021-05-12T12:49:50.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:49:50.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test6 1 1 2021-05-12T12:49:50.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2021-05-12T12:49:50.713 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test6' already exists 2021-05-12T12:49:50.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:49:51.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test6 size 2 2021-05-12T12:49:52.600 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 6 size to 2 2021-05-12T12:49:52.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:49:52.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test7 1 1 2021-05-12T12:49:52.611 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2021-05-12T12:49:53.196 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test7' already exists 2021-05-12T12:49:53.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:49:54.207 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test7 size 2 2021-05-12T12:49:54.987 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 7 size to 2 2021-05-12T12:49:54.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:49:54.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test8 1 1 2021-05-12T12:49:54.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2021-05-12T12:49:55.747 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test8' already exists 2021-05-12T12:49:55.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:49:56.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test8 size 2 2021-05-12T12:49:57.590 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 8 size to 2 2021-05-12T12:49:57.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:49:57.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test9 1 1 2021-05-12T12:49:57.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2021-05-12T12:49:58.135 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test9' already exists 2021-05-12T12:49:58.144 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:49:59.145 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test9 size 2 2021-05-12T12:50:00.162 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 9 size to 2 2021-05-12T12:50:00.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:00.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test10 1 1 2021-05-12T12:50:00.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2021-05-12T12:50:00.975 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test10' already exists 2021-05-12T12:50:00.984 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:01.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test10 size 2 2021-05-12T12:50:02.716 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 10 size to 2 2021-05-12T12:50:02.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: sleep 5 2021-05-12T12:50:07.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:65: TEST_recovery_priority: wait_for_clean 2021-05-12T12:50:07.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:50:07.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:50:07.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:50:07.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:50:07.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:50:07.729 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:50:07.729 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:50:07.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:50:07.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:50:07.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:50:07.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:50:07.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:50:07.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:50:07.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:50:07.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:50:08.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:50:08.531 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:50:08.531 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:50:08.532 INFO:tasks.workunit.client.0.smithi191.stderr:3 2021-05-12T12:50:08.532 INFO:tasks.workunit.client.0.smithi191.stderr:4' 2021-05-12T12:50:08.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:50:08.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:08.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:50:08.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836497 2021-05-12T12:50:08.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836497 2021-05-12T12:50:08.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497' 2021-05-12T12:50:08.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:08.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:50:08.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705680 2021-05-12T12:50:08.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705680 2021-05-12T12:50:08.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-38654705680' 2021-05-12T12:50:08.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:08.798 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:50:08.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542157 2021-05-12T12:50:08.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542157 2021-05-12T12:50:08.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542157' 2021-05-12T12:50:08.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:08.927 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:50:09.052 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=81604378635 2021-05-12T12:50:09.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 81604378635 2021-05-12T12:50:09.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542157 3-81604378635' 2021-05-12T12:50:09.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:09.053 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:50:09.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=103079215113 2021-05-12T12:50:09.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 103079215113 2021-05-12T12:50:09.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-60129542157 3-81604378635 4-103079215113' 2021-05-12T12:50:09.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:09.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836497 2021-05-12T12:50:09.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:09.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:50:09.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836497 2021-05-12T12:50:09.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:09.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836497 2021-05-12T12:50:09.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2021-05-12T12:50:09.184 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836497 2021-05-12T12:50:09.185 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:50:09.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836497 -lt 21474836497 2021-05-12T12:50:09.541 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:09.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705680 2021-05-12T12:50:09.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:09.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:50:09.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705680 2021-05-12T12:50:09.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:09.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705680 2021-05-12T12:50:09.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2021-05-12T12:50:09.545 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 38654705680 2021-05-12T12:50:09.545 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:50:09.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705680 -lt 38654705680 2021-05-12T12:50:09.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:09.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542157 2021-05-12T12:50:09.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:09.912 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:50:09.912 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542157 2021-05-12T12:50:09.913 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:09.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542157 2021-05-12T12:50:09.914 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2021-05-12T12:50:09.914 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542157 2021-05-12T12:50:09.914 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:50:10.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542157 -lt 60129542157 2021-05-12T12:50:10.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:10.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378635 2021-05-12T12:50:10.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:10.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:50:10.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-81604378635 2021-05-12T12:50:10.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:10.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=81604378635 2021-05-12T12:50:10.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 81604378635' 2021-05-12T12:50:10.297 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 81604378635 2021-05-12T12:50:10.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:50:10.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 81604378635 -lt 81604378635 2021-05-12T12:50:10.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:10.667 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215113 2021-05-12T12:50:10.668 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:10.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:50:10.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-103079215113 2021-05-12T12:50:10.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:10.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=103079215113 2021-05-12T12:50:10.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 103079215113' 2021-05-12T12:50:10.670 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.4 seq 103079215113 2021-05-12T12:50:10.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:50:11.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 103079215112 -lt 103079215113 2021-05-12T12:50:11.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:50:12.043 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:50:12.043 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:50:12.410 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 103079215113 -lt 103079215113 2021-05-12T12:50:12.411 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:50:12.411 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:50:12.411 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:50:12.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 10 == 0 2021-05-12T12:50:12.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:50:12.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:50:12.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:50:12.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:50:12.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:50:12.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:50:12.889 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:50:13.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=10 2021-05-12T12:50:13.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:50:13.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:50:13.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:50:13.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 10 = 10 2021-05-12T12:50:13.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:50:13.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:50:13.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:67: TEST_recovery_priority: ceph pg dump pgs 2021-05-12T12:50:14.050 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:50:14.050 INFO:tasks.workunit.client.0.smithi191.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:03.062513+0000 0'0 81:20 [2,1] 2 [2,1] 2 0'0 2021-05-12T12:50:00.558267+0000 0'0 2021-05-12T12:50:00.558267+0000 0 2021-05-12T12:50:14.051 INFO:tasks.workunit.client.0.smithi191.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:00.944452+0000 0'0 80:26 [1,2] 1 [1,2] 1 0'0 2021-05-12T12:49:57.937797+0000 0'0 2021-05-12T12:49:57.937797+0000 0 2021-05-12T12:50:14.051 INFO:tasks.workunit.client.0.smithi191.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:49:58.346398+0000 0'0 78:19 [3,4] 3 [3,4] 3 0'0 2021-05-12T12:49:55.391049+0000 0'0 2021-05-12T12:49:55.391049+0000 0 2021-05-12T12:50:14.051 INFO:tasks.workunit.client.0.smithi191.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:49:56.436685+0000 0'0 80:36 [1,4] 1 [1,4] 1 0'0 2021-05-12T12:49:52.968683+0000 0'0 2021-05-12T12:49:52.968683+0000 0 2021-05-12T12:50:14.051 INFO:tasks.workunit.client.0.smithi191.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:49:43.348720+0000 0'0 78:51 [3,1] 3 [3,1] 3 0'0 2021-05-12T12:49:40.888353+0000 0'0 2021-05-12T12:49:40.888353+0000 0 2021-05-12T12:50:14.052 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:49:40.767156+0000 0'0 80:61 [1,0] 1 [1,0] 1 0'0 2021-05-12T12:49:38.387886+0000 0'0 2021-05-12T12:49:38.387886+0000 0 2021-05-12T12:50:14.052 INFO:tasks.workunit.client.0.smithi191.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:49:45.812838+0000 0'0 80:45 [1,2] 1 [1,2] 1 0'0 2021-05-12T12:49:43.135094+0000 0'0 2021-05-12T12:49:43.135094+0000 0 2021-05-12T12:50:14.052 INFO:tasks.workunit.client.0.smithi191.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:49:48.437639+0000 0'0 79:39 [4,0] 4 [4,0] 4 0'0 2021-05-12T12:49:45.533040+0000 0'0 2021-05-12T12:49:45.533040+0000 0 2021-05-12T12:50:14.052 INFO:tasks.workunit.client.0.smithi191.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:49:50.399428+0000 0'0 78:39 [3,4] 3 [3,4] 3 0'0 2021-05-12T12:49:47.983199+0000 0'0 2021-05-12T12:49:47.983199+0000 0 2021-05-12T12:50:14.053 INFO:tasks.workunit.client.0.smithi191.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:49:53.768339+0000 0'0 79:35 [4,2] 4 [4,2] 4 0'0 2021-05-12T12:49:50.465889+0000 0'0 2021-05-12T12:49:50.465889+0000 0 2021-05-12T12:50:14.053 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:50:14.053 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:50:14.054 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:50:14.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:71: TEST_recovery_priority: local PG1 2021-05-12T12:50:14.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:72: TEST_recovery_priority: local POOLNUM1 2021-05-12T12:50:14.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:73: TEST_recovery_priority: local pool1 2021-05-12T12:50:14.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local chk_osd1_1 2021-05-12T12:50:14.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local chk_osd1_2 2021-05-12T12:50:14.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local PG2 2021-05-12T12:50:14.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local POOLNUM2 2021-05-12T12:50:14.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:79: TEST_recovery_priority: local pool2 2021-05-12T12:50:14.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local chk_osd2 2021-05-12T12:50:14.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local PG3 2021-05-12T12:50:14.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local POOLNUM3 2021-05-12T12:50:14.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:84: TEST_recovery_priority: local pool3 2021-05-12T12:50:14.066 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: seq 1 10 2021-05-12T12:50:14.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:14.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 1.0 --format=json 2021-05-12T12:50:14.066 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:14.415 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:14.415 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-05-12T12:50:14.416 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:14.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2021-05-12T12:50:14.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z '' ']' 2021-05-12T12:50:14.417 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: PG1=1.0 2021-05-12T12:50:14.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: POOLNUM1=1 2021-05-12T12:50:14.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:95: TEST_recovery_priority: pool1=test1 2021-05-12T12:50:14.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: chk_osd1_1=1 2021-05-12T12:50:14.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: chk_osd1_2=0 2021-05-12T12:50:14.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:14.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 2.0 --format=json 2021-05-12T12:50:14.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:14.806 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:14.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2021-05-12T12:50:14.807 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:14.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=1 2021-05-12T12:50:14.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:14.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2021-05-12T12:50:14.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2021-05-12T12:50:14.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:14.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 3.0 --format=json 2021-05-12T12:50:14.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:15.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:15.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-05-12T12:50:15.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:15.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2021-05-12T12:50:15.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:15.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2021-05-12T12:50:15.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: PG2=3.0 2021-05-12T12:50:15.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: POOLNUM2=3 2021-05-12T12:50:15.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:102: TEST_recovery_priority: pool2=test3 2021-05-12T12:50:15.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: chk_osd2=2 2021-05-12T12:50:15.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:15.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 4.0 --format=json 2021-05-12T12:50:15.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:15.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:15.544 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2021-05-12T12:50:15.544 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:15.545 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2021-05-12T12:50:15.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:15.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2021-05-12T12:50:15.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2021-05-12T12:50:15.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:15.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 5.0 --format=json 2021-05-12T12:50:15.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:15.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:15.924 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2021-05-12T12:50:15.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:15.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2021-05-12T12:50:15.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:15.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2021-05-12T12:50:15.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-05-12T12:50:15.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:15.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 6.0 --format=json 2021-05-12T12:50:15.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:16.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:16.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2021-05-12T12:50:16.288 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:16.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2021-05-12T12:50:16.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:16.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2021-05-12T12:50:16.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2021-05-12T12:50:16.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:16.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 7.0 --format=json 2021-05-12T12:50:16.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:16.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:16.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-05-12T12:50:16.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:16.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2021-05-12T12:50:16.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:16.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2021-05-12T12:50:16.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-05-12T12:50:16.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: PG3=7.0 2021-05-12T12:50:16.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: POOLNUM3=7 2021-05-12T12:50:16.648 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:108: TEST_recovery_priority: pool3=test7 2021-05-12T12:50:16.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: break 2021-05-12T12:50:16.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2021-05-12T12:50:16.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:114: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2021-05-12T12:50:16.649 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: seq 1 10 2021-05-12T12:50:16.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:16.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2021-05-12T12:50:16.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:16.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2021-05-12T12:50:16.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test2 2021-05-12T12:50:16.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2021-05-12T12:50:16.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-05-12T12:50:17.145 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test2' does not exist 2021-05-12T12:50:17.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:17.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2021-05-12T12:50:17.155 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:17.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2021-05-12T12:50:17.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test4 2021-05-12T12:50:17.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2021-05-12T12:50:17.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2021-05-12T12:50:17.747 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test4' does not exist 2021-05-12T12:50:17.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:17.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2021-05-12T12:50:17.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test5 2021-05-12T12:50:17.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2021-05-12T12:50:17.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2021-05-12T12:50:18.283 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test5' does not exist 2021-05-12T12:50:18.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:18.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2021-05-12T12:50:18.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test6 2021-05-12T12:50:18.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2021-05-12T12:50:18.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2021-05-12T12:50:18.909 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test6' does not exist 2021-05-12T12:50:18.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:18.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2021-05-12T12:50:18.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:18.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2021-05-12T12:50:18.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test8 2021-05-12T12:50:18.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2021-05-12T12:50:18.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2021-05-12T12:50:19.632 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test8' does not exist 2021-05-12T12:50:19.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:19.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2021-05-12T12:50:19.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test9 2021-05-12T12:50:19.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2021-05-12T12:50:19.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2021-05-12T12:50:20.262 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test9' does not exist 2021-05-12T12:50:20.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:20.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2021-05-12T12:50:20.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test10 2021-05-12T12:50:20.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2021-05-12T12:50:20.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2021-05-12T12:50:20.770 INFO:tasks.workunit.client.0.smithi191.stderr:pool 'test10' does not exist 2021-05-12T12:50:20.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:128: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2021-05-12T12:50:21.654 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 3 size to 1 2021-05-12T12:50:21.663 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:129: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2021-05-12T12:50:22.319 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 7 size to 1 2021-05-12T12:50:22.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:130: TEST_recovery_priority: wait_for_clean 2021-05-12T12:50:22.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:50:22.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:50:22.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:50:22.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:50:22.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:50:22.330 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:50:22.331 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:50:22.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:50:22.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:50:22.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:50:22.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:50:22.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:50:22.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:50:22.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:50:22.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:50:22.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:50:22.831 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:50:22.832 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:50:22.832 INFO:tasks.workunit.client.0.smithi191.stderr:3 2021-05-12T12:50:22.832 INFO:tasks.workunit.client.0.smithi191.stderr:4' 2021-05-12T12:50:22.832 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:50:22.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:22.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:50:22.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836501 2021-05-12T12:50:22.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836501 2021-05-12T12:50:22.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836501' 2021-05-12T12:50:22.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:22.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:50:23.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705685 2021-05-12T12:50:23.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705685 2021-05-12T12:50:23.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836501 1-38654705685' 2021-05-12T12:50:23.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:23.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:50:23.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542162 2021-05-12T12:50:23.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542162 2021-05-12T12:50:23.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836501 1-38654705685 2-60129542162' 2021-05-12T12:50:23.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:23.214 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:50:23.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=81604378640 2021-05-12T12:50:23.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 81604378640 2021-05-12T12:50:23.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836501 1-38654705685 2-60129542162 3-81604378640' 2021-05-12T12:50:23.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:23.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:50:23.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=103079215118 2021-05-12T12:50:23.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 103079215118 2021-05-12T12:50:23.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836501 1-38654705685 2-60129542162 3-81604378640 4-103079215118' 2021-05-12T12:50:23.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:23.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836501 2021-05-12T12:50:23.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:23.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:50:23.473 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836501 2021-05-12T12:50:23.473 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:23.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836501 2021-05-12T12:50:23.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2021-05-12T12:50:23.474 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836501 2021-05-12T12:50:23.474 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:50:23.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836501 -lt 21474836501 2021-05-12T12:50:23.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:23.852 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705685 2021-05-12T12:50:23.852 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:23.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:50:23.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705685 2021-05-12T12:50:23.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:23.854 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705685 2021-05-12T12:50:23.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705685' 2021-05-12T12:50:23.855 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 38654705685 2021-05-12T12:50:23.855 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:50:24.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705685 -lt 38654705685 2021-05-12T12:50:24.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:24.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542162 2021-05-12T12:50:24.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:24.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:50:24.218 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542162 2021-05-12T12:50:24.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:24.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542162 2021-05-12T12:50:24.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2021-05-12T12:50:24.219 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542162 2021-05-12T12:50:24.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:50:24.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542160 -lt 60129542162 2021-05-12T12:50:24.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:50:25.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:50:25.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:50:25.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542162 -lt 60129542162 2021-05-12T12:50:25.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:25.958 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378640 2021-05-12T12:50:25.959 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:25.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:50:25.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-81604378640 2021-05-12T12:50:25.960 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:25.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=81604378640 2021-05-12T12:50:25.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 81604378640' 2021-05-12T12:50:25.961 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 81604378640 2021-05-12T12:50:25.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:50:26.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 81604378640 -lt 81604378640 2021-05-12T12:50:26.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:26.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215118 2021-05-12T12:50:26.432 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:26.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:50:26.433 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-103079215118 2021-05-12T12:50:26.434 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:26.434 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=103079215118 2021-05-12T12:50:26.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 103079215118' 2021-05-12T12:50:26.435 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.4 seq 103079215118 2021-05-12T12:50:26.435 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:50:26.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 103079215118 -lt 103079215118 2021-05-12T12:50:26.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:50:26.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:50:26.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:50:27.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 3 == 0 2021-05-12T12:50:27.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:50:27.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:50:27.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:50:27.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:50:27.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:50:27.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:50:27.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:50:27.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=3 2021-05-12T12:50:27.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:50:27.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:50:27.620 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:50:28.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 3 = 3 2021-05-12T12:50:28.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:50:28.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:50:28.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2021-05-12T12:50:28.135 INFO:tasks.workunit.client.0.smithi191.stderr:10+0 records in 2021-05-12T12:50:28.136 INFO:tasks.workunit.client.0.smithi191.stderr:10+0 records out 2021-05-12T12:50:28.136 INFO:tasks.workunit.client.0.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.045454 s, 231 MB/s 2021-05-12T12:50:28.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: p=1 2021-05-12T12:50:28.137 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-05-12T12:50:28.137 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-05-12T12:50:28.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:28.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2021-05-12T12:50:28.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:28.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2021-05-12T12:50:29.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:29.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2021-05-12T12:50:29.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:29.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2021-05-12T12:50:31.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:31.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2021-05-12T12:50:31.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:31.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2021-05-12T12:50:32.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:32.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2021-05-12T12:50:32.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:32.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2021-05-12T12:50:33.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:33.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2021-05-12T12:50:34.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:34.515 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2021-05-12T12:50:35.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:35.050 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2021-05-12T12:50:35.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:35.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2021-05-12T12:50:36.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:36.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2021-05-12T12:50:37.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:37.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2021-05-12T12:50:37.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:37.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2021-05-12T12:50:38.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:38.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2021-05-12T12:50:39.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:39.027 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2021-05-12T12:50:40.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:40.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2021-05-12T12:50:40.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:40.923 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2021-05-12T12:50:41.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:41.559 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2021-05-12T12:50:42.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:42.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2021-05-12T12:50:43.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:43.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2021-05-12T12:50:43.761 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:43.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2021-05-12T12:50:44.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:44.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2021-05-12T12:50:46.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:46.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2021-05-12T12:50:46.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:46.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2021-05-12T12:50:47.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:47.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2021-05-12T12:50:48.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:48.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2021-05-12T12:50:49.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:49.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2021-05-12T12:50:49.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:49.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2021-05-12T12:50:50.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:50.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2021-05-12T12:50:50.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:50.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2021-05-12T12:50:52.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:52.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2021-05-12T12:50:53.020 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:53.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2021-05-12T12:50:53.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:53.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2021-05-12T12:50:54.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:54.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2021-05-12T12:50:55.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:55.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2021-05-12T12:50:55.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:55.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2021-05-12T12:50:56.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:56.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2021-05-12T12:50:57.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:57.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2021-05-12T12:50:58.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:58.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2021-05-12T12:50:58.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:58.897 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2021-05-12T12:50:59.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:59.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2021-05-12T12:51:00.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:00.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2021-05-12T12:51:01.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:01.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2021-05-12T12:51:01.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:01.929 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2021-05-12T12:51:02.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:02.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2021-05-12T12:51:04.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:04.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2021-05-12T12:51:05.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:05.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2021-05-12T12:51:05.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:05.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2021-05-12T12:51:06.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:06.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2021-05-12T12:51:07.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:07.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2021-05-12T12:51:08.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:08.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2021-05-12T12:51:08.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:08.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2021-05-12T12:51:09.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:09.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2021-05-12T12:51:10.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:10.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2021-05-12T12:51:11.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:11.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2021-05-12T12:51:11.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:11.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2021-05-12T12:51:12.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:12.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2021-05-12T12:51:13.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:13.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2021-05-12T12:51:14.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:14.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2021-05-12T12:51:14.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:14.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2021-05-12T12:51:15.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:15.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2021-05-12T12:51:16.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:16.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2021-05-12T12:51:17.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:17.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2021-05-12T12:51:17.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:17.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2021-05-12T12:51:18.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:18.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2021-05-12T12:51:19.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:19.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2021-05-12T12:51:20.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:20.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2021-05-12T12:51:20.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:20.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2021-05-12T12:51:21.432 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:21.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2021-05-12T12:51:23.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:23.695 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2021-05-12T12:51:24.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:24.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2021-05-12T12:51:25.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:25.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2021-05-12T12:51:25.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:25.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2021-05-12T12:51:26.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:26.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2021-05-12T12:51:26.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:26.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2021-05-12T12:51:27.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:27.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2021-05-12T12:51:28.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:28.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2021-05-12T12:51:29.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:29.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2021-05-12T12:51:29.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:29.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2021-05-12T12:51:30.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:30.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2021-05-12T12:51:31.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:31.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2021-05-12T12:51:32.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:32.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2021-05-12T12:51:32.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:32.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2021-05-12T12:51:33.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:33.514 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2021-05-12T12:51:34.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:34.798 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2021-05-12T12:51:35.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:35.305 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2021-05-12T12:51:35.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:35.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2021-05-12T12:51:36.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:36.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2021-05-12T12:51:37.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:37.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2021-05-12T12:51:38.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:38.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2021-05-12T12:51:38.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:38.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2021-05-12T12:51:39.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:39.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2021-05-12T12:51:41.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:41.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2021-05-12T12:51:43.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:43.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2021-05-12T12:51:44.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:44.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2021-05-12T12:51:44.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:44.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2021-05-12T12:51:45.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:45.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2021-05-12T12:51:46.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:46.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2021-05-12T12:51:47.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:47.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2021-05-12T12:51:47.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:47.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2021-05-12T12:51:48.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:48.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2021-05-12T12:51:50.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:50.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2021-05-12T12:51:51.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:51.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2021-05-12T12:51:51.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:51.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2021-05-12T12:51:52.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:52.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2021-05-12T12:51:53.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:53.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2021-05-12T12:51:54.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:54.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2021-05-12T12:51:54.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:54.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2021-05-12T12:51:56.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:56.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2021-05-12T12:51:56.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:56.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2021-05-12T12:51:57.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:57.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2021-05-12T12:51:57.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:57.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2021-05-12T12:51:59.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:59.040 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2021-05-12T12:51:59.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:59.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2021-05-12T12:52:00.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:00.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2021-05-12T12:52:00.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:00.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2021-05-12T12:52:03.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:03.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2021-05-12T12:52:03.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:03.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2021-05-12T12:52:04.796 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:04.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2021-05-12T12:52:05.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:05.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2021-05-12T12:52:06.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:06.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2021-05-12T12:52:06.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:06.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2021-05-12T12:52:07.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:07.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2021-05-12T12:52:08.490 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:08.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2021-05-12T12:52:09.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:09.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2021-05-12T12:52:09.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:09.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2021-05-12T12:52:10.888 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:10.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2021-05-12T12:52:11.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:11.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2021-05-12T12:52:12.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:12.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2021-05-12T12:52:12.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:12.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2021-05-12T12:52:13.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:13.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2021-05-12T12:52:14.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:14.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2021-05-12T12:52:15.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:15.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2021-05-12T12:52:15.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:15.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2021-05-12T12:52:16.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:16.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2021-05-12T12:52:17.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:17.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2021-05-12T12:52:18.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:18.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2021-05-12T12:52:18.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:18.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2021-05-12T12:52:19.579 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:19.580 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2021-05-12T12:52:21.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:21.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2021-05-12T12:52:22.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:22.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2021-05-12T12:52:23.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:23.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2021-05-12T12:52:23.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:23.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2021-05-12T12:52:24.659 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:24.660 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2021-05-12T12:52:25.246 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:25.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2021-05-12T12:52:26.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:26.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2021-05-12T12:52:27.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:27.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2021-05-12T12:52:27.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:27.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2021-05-12T12:52:29.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:29.146 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2021-05-12T12:52:29.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:29.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2021-05-12T12:52:30.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:30.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2021-05-12T12:52:30.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:30.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2021-05-12T12:52:31.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:31.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2021-05-12T12:52:32.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:32.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2021-05-12T12:52:33.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:33.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2021-05-12T12:52:33.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:33.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2021-05-12T12:52:34.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:34.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2021-05-12T12:52:35.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:35.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2021-05-12T12:52:36.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:36.491 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2021-05-12T12:52:37.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:37.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2021-05-12T12:52:38.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:38.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2021-05-12T12:52:38.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:38.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2021-05-12T12:52:39.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:39.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2021-05-12T12:52:41.473 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:41.474 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2021-05-12T12:52:42.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:42.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2021-05-12T12:52:42.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:42.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2021-05-12T12:52:43.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:43.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2021-05-12T12:52:44.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:44.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2021-05-12T12:52:44.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:44.939 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2021-05-12T12:52:45.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:45.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2021-05-12T12:52:46.169 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:46.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2021-05-12T12:52:47.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:47.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2021-05-12T12:52:48.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:48.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2021-05-12T12:52:48.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:48.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2021-05-12T12:52:49.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:49.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2021-05-12T12:52:50.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:50.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2021-05-12T12:52:51.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:51.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2021-05-12T12:52:51.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:51.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2021-05-12T12:52:52.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:52.316 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2021-05-12T12:52:53.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:53.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2021-05-12T12:52:54.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:54.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2021-05-12T12:52:54.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:54.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2021-05-12T12:52:55.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:55.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2021-05-12T12:52:56.772 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:56.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2021-05-12T12:52:57.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:57.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2021-05-12T12:52:57.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:57.927 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2021-05-12T12:53:00.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:00.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2021-05-12T12:53:01.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:01.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2021-05-12T12:53:02.517 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:02.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2021-05-12T12:53:03.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:03.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2021-05-12T12:53:03.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:03.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2021-05-12T12:53:04.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:04.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2021-05-12T12:53:05.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:05.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2021-05-12T12:53:06.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:06.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2021-05-12T12:53:06.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:06.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2021-05-12T12:53:07.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:07.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2021-05-12T12:53:08.714 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:08.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2021-05-12T12:53:09.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:09.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2021-05-12T12:53:09.808 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 1 + 1 2021-05-12T12:53:09.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=2 2021-05-12T12:53:09.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-05-12T12:53:09.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-05-12T12:53:09.810 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:09.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2021-05-12T12:53:10.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:10.151 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2021-05-12T12:53:10.477 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:10.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2021-05-12T12:53:11.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:11.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2021-05-12T12:53:11.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:11.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2021-05-12T12:53:12.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:12.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2021-05-12T12:53:13.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:13.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2021-05-12T12:53:13.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:13.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2021-05-12T12:53:14.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:14.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2021-05-12T12:53:14.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:14.795 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2021-05-12T12:53:15.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:15.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2021-05-12T12:53:15.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:15.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2021-05-12T12:53:16.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:16.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2021-05-12T12:53:16.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:16.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2021-05-12T12:53:16.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:16.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2021-05-12T12:53:17.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:17.247 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2021-05-12T12:53:18.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:18.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2021-05-12T12:53:18.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:18.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2021-05-12T12:53:18.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:18.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2021-05-12T12:53:19.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:19.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2021-05-12T12:53:19.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:19.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2021-05-12T12:53:19.948 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:19.949 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2021-05-12T12:53:20.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:20.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2021-05-12T12:53:21.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:21.143 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2021-05-12T12:53:21.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:21.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2021-05-12T12:53:22.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:22.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2021-05-12T12:53:22.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:22.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2021-05-12T12:53:22.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:22.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2021-05-12T12:53:22.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:22.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2021-05-12T12:53:23.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:23.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2021-05-12T12:53:24.281 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:24.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2021-05-12T12:53:24.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:24.591 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2021-05-12T12:53:24.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:24.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2021-05-12T12:53:25.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:25.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2021-05-12T12:53:25.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:25.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2021-05-12T12:53:26.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:26.454 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2021-05-12T12:53:27.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:27.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2021-05-12T12:53:28.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:28.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2021-05-12T12:53:28.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:28.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2021-05-12T12:53:28.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:28.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2021-05-12T12:53:29.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:29.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2021-05-12T12:53:29.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:29.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2021-05-12T12:53:30.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:30.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2021-05-12T12:53:30.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:30.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2021-05-12T12:53:31.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:31.029 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2021-05-12T12:53:31.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:31.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2021-05-12T12:53:31.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:31.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2021-05-12T12:53:32.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:32.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2021-05-12T12:53:32.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:32.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2021-05-12T12:53:33.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:33.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2021-05-12T12:53:33.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:33.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2021-05-12T12:53:34.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:34.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2021-05-12T12:53:34.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:34.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2021-05-12T12:53:34.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:34.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2021-05-12T12:53:35.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:35.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2021-05-12T12:53:35.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:35.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2021-05-12T12:53:36.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:36.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2021-05-12T12:53:36.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:36.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2021-05-12T12:53:37.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:37.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2021-05-12T12:53:38.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:38.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2021-05-12T12:53:38.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:38.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2021-05-12T12:53:38.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:38.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2021-05-12T12:53:39.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:39.543 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2021-05-12T12:53:39.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:39.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2021-05-12T12:53:40.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:40.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2021-05-12T12:53:40.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:40.529 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2021-05-12T12:53:40.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:40.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2021-05-12T12:53:41.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:41.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2021-05-12T12:53:41.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:41.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2021-05-12T12:53:42.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:42.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2021-05-12T12:53:42.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:42.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2021-05-12T12:53:43.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:43.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2021-05-12T12:53:43.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:43.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2021-05-12T12:53:43.954 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:43.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2021-05-12T12:53:44.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:44.289 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2021-05-12T12:53:44.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:44.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2021-05-12T12:53:45.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:45.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2021-05-12T12:53:45.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:45.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2021-05-12T12:53:46.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:46.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2021-05-12T12:53:46.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:46.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2021-05-12T12:53:46.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:46.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2021-05-12T12:53:47.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:47.171 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2021-05-12T12:53:48.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:48.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2021-05-12T12:53:49.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:49.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2021-05-12T12:53:50.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:50.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2021-05-12T12:53:50.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:50.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2021-05-12T12:53:51.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:51.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2021-05-12T12:53:51.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:51.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2021-05-12T12:53:52.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:52.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2021-05-12T12:53:52.340 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:52.341 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2021-05-12T12:53:52.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:52.725 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2021-05-12T12:53:53.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:53.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2021-05-12T12:53:53.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:53.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2021-05-12T12:53:53.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:53.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2021-05-12T12:53:54.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:54.913 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2021-05-12T12:53:55.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:55.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2021-05-12T12:53:55.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:55.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2021-05-12T12:53:56.024 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:56.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2021-05-12T12:53:56.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:56.359 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2021-05-12T12:53:56.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:56.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2021-05-12T12:53:57.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:57.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2021-05-12T12:53:58.204 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:58.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2021-05-12T12:53:58.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:58.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2021-05-12T12:53:58.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:58.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2021-05-12T12:53:59.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:59.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2021-05-12T12:53:59.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:59.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2021-05-12T12:54:01.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:01.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2021-05-12T12:54:01.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:01.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2021-05-12T12:54:02.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:02.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2021-05-12T12:54:03.018 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:03.018 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2021-05-12T12:54:03.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:03.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2021-05-12T12:54:04.053 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:04.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2021-05-12T12:54:04.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:04.379 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2021-05-12T12:54:04.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:04.688 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2021-05-12T12:54:04.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:04.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2021-05-12T12:54:05.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:05.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2021-05-12T12:54:05.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:05.758 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2021-05-12T12:54:06.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:06.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2021-05-12T12:54:07.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:07.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2021-05-12T12:54:07.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:07.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2021-05-12T12:54:07.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:07.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2021-05-12T12:54:08.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:08.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2021-05-12T12:54:08.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:08.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2021-05-12T12:54:08.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:08.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2021-05-12T12:54:09.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:09.150 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2021-05-12T12:54:10.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:10.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2021-05-12T12:54:10.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:10.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2021-05-12T12:54:10.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:10.786 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2021-05-12T12:54:11.103 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:11.104 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2021-05-12T12:54:11.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:11.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2021-05-12T12:54:12.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:12.732 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2021-05-12T12:54:13.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:13.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2021-05-12T12:54:13.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:13.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2021-05-12T12:54:13.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:13.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2021-05-12T12:54:14.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:14.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2021-05-12T12:54:14.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:14.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2021-05-12T12:54:14.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:14.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2021-05-12T12:54:15.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:15.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2021-05-12T12:54:16.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:16.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2021-05-12T12:54:16.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:16.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2021-05-12T12:54:16.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:16.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2021-05-12T12:54:17.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:17.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2021-05-12T12:54:17.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:17.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2021-05-12T12:54:17.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:17.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2021-05-12T12:54:18.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:18.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2021-05-12T12:54:19.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:19.249 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2021-05-12T12:54:19.616 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:19.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2021-05-12T12:54:19.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:19.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2021-05-12T12:54:20.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:20.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2021-05-12T12:54:20.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:20.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2021-05-12T12:54:20.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:20.928 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2021-05-12T12:54:21.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:21.229 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2021-05-12T12:54:21.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:21.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2021-05-12T12:54:23.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:23.400 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2021-05-12T12:54:23.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:24.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2021-05-12T12:54:24.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:24.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2021-05-12T12:54:24.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:24.553 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2021-05-12T12:54:25.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:25.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2021-05-12T12:54:25.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:25.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2021-05-12T12:54:25.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:25.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2021-05-12T12:54:26.257 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:26.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2021-05-12T12:54:26.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:26.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2021-05-12T12:54:26.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:26.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2021-05-12T12:54:27.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:27.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2021-05-12T12:54:27.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:27.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2021-05-12T12:54:28.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:28.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2021-05-12T12:54:29.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:29.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2021-05-12T12:54:29.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:29.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2021-05-12T12:54:29.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:29.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2021-05-12T12:54:30.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:30.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2021-05-12T12:54:30.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:30.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2021-05-12T12:54:30.627 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:30.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2021-05-12T12:54:31.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:31.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2021-05-12T12:54:31.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:31.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2021-05-12T12:54:32.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:32.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2021-05-12T12:54:32.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:32.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2021-05-12T12:54:33.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:33.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2021-05-12T12:54:34.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:34.662 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2021-05-12T12:54:35.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:35.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2021-05-12T12:54:35.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:35.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2021-05-12T12:54:36.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:36.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2021-05-12T12:54:36.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:36.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2021-05-12T12:54:36.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:36.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2021-05-12T12:54:37.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:37.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2021-05-12T12:54:37.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:37.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2021-05-12T12:54:38.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:38.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2021-05-12T12:54:38.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:38.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2021-05-12T12:54:38.980 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:38.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2021-05-12T12:54:39.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:39.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2021-05-12T12:54:39.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:39.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2021-05-12T12:54:40.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:40.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2021-05-12T12:54:41.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:41.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2021-05-12T12:54:41.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:41.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2021-05-12T12:54:41.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:41.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2021-05-12T12:54:42.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:42.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2021-05-12T12:54:42.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:42.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2021-05-12T12:54:42.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:42.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2021-05-12T12:54:43.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:43.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2021-05-12T12:54:44.184 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:44.185 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2021-05-12T12:54:44.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:44.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2021-05-12T12:54:45.128 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 2 + 1 2021-05-12T12:54:45.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=3 2021-05-12T12:54:45.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-05-12T12:54:45.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-05-12T12:54:45.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:45.131 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2021-05-12T12:54:45.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:45.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2021-05-12T12:54:45.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:45.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2021-05-12T12:54:46.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:46.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2021-05-12T12:54:46.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:46.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2021-05-12T12:54:47.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:47.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2021-05-12T12:54:47.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:47.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2021-05-12T12:54:48.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:48.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2021-05-12T12:54:48.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:48.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2021-05-12T12:54:48.718 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:48.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2021-05-12T12:54:50.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:50.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2021-05-12T12:54:50.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:50.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2021-05-12T12:54:51.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:51.237 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2021-05-12T12:54:51.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:51.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2021-05-12T12:54:51.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:51.998 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2021-05-12T12:54:52.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:52.741 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2021-05-12T12:54:53.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:53.100 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2021-05-12T12:54:53.475 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:53.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2021-05-12T12:54:54.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:54.044 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2021-05-12T12:54:54.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:54.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2021-05-12T12:54:54.687 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:54.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2021-05-12T12:54:55.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:55.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2021-05-12T12:54:55.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:55.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2021-05-12T12:54:56.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:56.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2021-05-12T12:54:57.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:57.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2021-05-12T12:54:58.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:58.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2021-05-12T12:54:58.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:58.794 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2021-05-12T12:54:59.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:59.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2021-05-12T12:55:00.025 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:00.026 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2021-05-12T12:55:00.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:00.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2021-05-12T12:55:00.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:00.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2021-05-12T12:55:01.002 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:01.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2021-05-12T12:55:01.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:01.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2021-05-12T12:55:02.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:02.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2021-05-12T12:55:02.564 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:02.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2021-05-12T12:55:02.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:02.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2021-05-12T12:55:03.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:03.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2021-05-12T12:55:03.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:03.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2021-05-12T12:55:04.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:04.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2021-05-12T12:55:04.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:04.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2021-05-12T12:55:05.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:05.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2021-05-12T12:55:05.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:05.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2021-05-12T12:55:06.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:06.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2021-05-12T12:55:06.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:06.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2021-05-12T12:55:06.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:06.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2021-05-12T12:55:07.075 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:07.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2021-05-12T12:55:07.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:07.452 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2021-05-12T12:55:09.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:09.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2021-05-12T12:55:09.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:09.708 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2021-05-12T12:55:10.076 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:10.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2021-05-12T12:55:10.393 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:10.394 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2021-05-12T12:55:11.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:11.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2021-05-12T12:55:11.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:11.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2021-05-12T12:55:11.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:11.797 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2021-05-12T12:55:12.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:12.148 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2021-05-12T12:55:12.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:12.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2021-05-12T12:55:12.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:12.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2021-05-12T12:55:13.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:13.118 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2021-05-12T12:55:13.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:13.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2021-05-12T12:55:14.503 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:14.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2021-05-12T12:55:14.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:14.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2021-05-12T12:55:15.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:15.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2021-05-12T12:55:15.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:15.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2021-05-12T12:55:15.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:15.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2021-05-12T12:55:16.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:16.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2021-05-12T12:55:16.601 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:16.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2021-05-12T12:55:17.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:17.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2021-05-12T12:55:17.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:17.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2021-05-12T12:55:18.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:18.280 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2021-05-12T12:55:18.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:18.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2021-05-12T12:55:19.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:19.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2021-05-12T12:55:20.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:20.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2021-05-12T12:55:21.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:21.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2021-05-12T12:55:21.629 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:21.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2021-05-12T12:55:22.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:22.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2021-05-12T12:55:22.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:22.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2021-05-12T12:55:22.632 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:22.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2021-05-12T12:55:22.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:22.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2021-05-12T12:55:23.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:23.760 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2021-05-12T12:55:24.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:24.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2021-05-12T12:55:24.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:24.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2021-05-12T12:55:24.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:24.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2021-05-12T12:55:25.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:25.147 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2021-05-12T12:55:25.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:25.540 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2021-05-12T12:55:25.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:25.900 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2021-05-12T12:55:26.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:26.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2021-05-12T12:55:27.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:27.051 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2021-05-12T12:55:27.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:27.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2021-05-12T12:55:27.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:27.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2021-05-12T12:55:28.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:28.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2021-05-12T12:55:28.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:28.456 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2021-05-12T12:55:28.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:28.899 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2021-05-12T12:55:29.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:29.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2021-05-12T12:55:30.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:30.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2021-05-12T12:55:30.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:30.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2021-05-12T12:55:31.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:31.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2021-05-12T12:55:31.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:31.647 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2021-05-12T12:55:32.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:32.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2021-05-12T12:55:32.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:32.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2021-05-12T12:55:33.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:33.149 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2021-05-12T12:55:33.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:33.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2021-05-12T12:55:33.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:33.985 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2021-05-12T12:55:34.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:34.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2021-05-12T12:55:34.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:34.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2021-05-12T12:55:35.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:35.005 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2021-05-12T12:55:35.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:35.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2021-05-12T12:55:36.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:36.283 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2021-05-12T12:55:36.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:36.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2021-05-12T12:55:36.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:36.934 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2021-05-12T12:55:37.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:37.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2021-05-12T12:55:37.619 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:37.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2021-05-12T12:55:37.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:37.996 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2021-05-12T12:55:39.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:39.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2021-05-12T12:55:40.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:40.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2021-05-12T12:55:40.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:40.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2021-05-12T12:55:40.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:40.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2021-05-12T12:55:41.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:41.021 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2021-05-12T12:55:42.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:42.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2021-05-12T12:55:42.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:42.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2021-05-12T12:55:43.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:43.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2021-05-12T12:55:43.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:43.560 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2021-05-12T12:55:43.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:43.962 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2021-05-12T12:55:44.287 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:44.288 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2021-05-12T12:55:45.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:45.105 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2021-05-12T12:55:45.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:45.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2021-05-12T12:55:45.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:45.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2021-05-12T12:55:46.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:46.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2021-05-12T12:55:46.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:46.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2021-05-12T12:55:46.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:46.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2021-05-12T12:55:47.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:47.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2021-05-12T12:55:47.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:47.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2021-05-12T12:55:48.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:48.431 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2021-05-12T12:55:48.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:48.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2021-05-12T12:55:49.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:49.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2021-05-12T12:55:49.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:49.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2021-05-12T12:55:49.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:49.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2021-05-12T12:55:50.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:50.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2021-05-12T12:55:50.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:50.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2021-05-12T12:55:51.463 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:51.464 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2021-05-12T12:55:51.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:51.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2021-05-12T12:55:52.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:52.215 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2021-05-12T12:55:52.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:52.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2021-05-12T12:55:53.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:53.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2021-05-12T12:55:53.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:53.661 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2021-05-12T12:55:54.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:54.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2021-05-12T12:55:54.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:54.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2021-05-12T12:55:55.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:55.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2021-05-12T12:55:55.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:55.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2021-05-12T12:55:56.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:56.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2021-05-12T12:55:56.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:56.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2021-05-12T12:55:56.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:56.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2021-05-12T12:55:57.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:57.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2021-05-12T12:55:58.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:58.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2021-05-12T12:55:58.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:58.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2021-05-12T12:55:58.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:58.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2021-05-12T12:55:59.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:59.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2021-05-12T12:55:59.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:59.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2021-05-12T12:56:00.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:00.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2021-05-12T12:56:00.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:00.836 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2021-05-12T12:56:01.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:01.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2021-05-12T12:56:01.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:01.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2021-05-12T12:56:01.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:01.973 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2021-05-12T12:56:02.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:02.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2021-05-12T12:56:02.599 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:02.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2021-05-12T12:56:03.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:03.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2021-05-12T12:56:04.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:04.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2021-05-12T12:56:06.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:06.348 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2021-05-12T12:56:06.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:06.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2021-05-12T12:56:06.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:06.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2021-05-12T12:56:07.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:07.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2021-05-12T12:56:07.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:07.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2021-05-12T12:56:08.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:08.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2021-05-12T12:56:08.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:08.304 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2021-05-12T12:56:08.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:08.655 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2021-05-12T12:56:09.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:09.782 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2021-05-12T12:56:10.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:10.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2021-05-12T12:56:10.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:10.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2021-05-12T12:56:10.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:10.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2021-05-12T12:56:11.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:11.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2021-05-12T12:56:11.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:11.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2021-05-12T12:56:11.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:11.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2021-05-12T12:56:12.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:12.214 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2021-05-12T12:56:12.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:12.565 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2021-05-12T12:56:13.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:13.558 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2021-05-12T12:56:13.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:13.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2021-05-12T12:56:14.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:14.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2021-05-12T12:56:14.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:14.536 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2021-05-12T12:56:14.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:14.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2021-05-12T12:56:16.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:16.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2021-05-12T12:56:16.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:16.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2021-05-12T12:56:17.041 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:17.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2021-05-12T12:56:17.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:17.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2021-05-12T12:56:17.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:17.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2021-05-12T12:56:18.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:18.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2021-05-12T12:56:18.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:18.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2021-05-12T12:56:18.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:18.713 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2021-05-12T12:56:19.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:19.022 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2021-05-12T12:56:20.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:20.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2021-05-12T12:56:20.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:20.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2021-05-12T12:56:20.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:20.734 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2021-05-12T12:56:21.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 3 + 1 2021-05-12T12:56:21.069 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=4 2021-05-12T12:56:21.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: get_not_primary test1 obj1-p1 2021-05-12T12:56:21.070 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local poolname=test1 2021-05-12T12:56:21.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local objectname=obj1-p1 2021-05-12T12:56:21.071 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: get_primary test1 obj1-p1 2021-05-12T12:56:21.071 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local poolname=test1 2021-05-12T12:56:21.071 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: local objectname=obj1-p1 2021-05-12T12:56:21.072 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T12:56:21.072 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: jq .acting_primary 2021-05-12T12:56:21.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local primary=1 2021-05-12T12:56:21.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T12:56:21.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1173: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-05-12T12:56:21.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: local otherosd=0 2021-05-12T12:56:21.807 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:145: TEST_recovery_priority: ceph pg dump pgs 2021-05-12T12:56:22.166 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:56:22.167 INFO:tasks.workunit.client.0.smithi191.stdout:7.0 192 0 0 0 0 2006974464 0 0 574 574 active+clean 2021-05-12T12:50:22.933492+0000 93'574 93:626 [1] 1 [1] 1 0'0 2021-05-12T12:49:52.968683+0000 0'0 2021-05-12T12:49:52.968683+0000 0 2021-05-12T12:56:22.167 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:49:40.767156+0000 93'600 93:670 [1,0] 1 [1,0] 1 0'0 2021-05-12T12:49:38.387886+0000 0'0 2021-05-12T12:49:38.387886+0000 0 2021-05-12T12:56:22.167 INFO:tasks.workunit.client.0.smithi191.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:50:22.967081+0000 93'600 93:661 [1] 1 [1] 1 0'0 2021-05-12T12:49:43.135094+0000 0'0 2021-05-12T12:49:43.135094+0000 0 2021-05-12T12:56:22.168 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:56:22.168 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:56:22.168 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:56:22.177 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: ERRORS=0 2021-05-12T12:56:22.178 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph osd set norecover 2021-05-12T12:56:22.921 INFO:tasks.workunit.client.0.smithi191.stderr:norecover is set 2021-05-12T12:56:22.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ceph osd set noout 2021-05-12T12:56:23.658 INFO:tasks.workunit.client.0.smithi191.stderr:noout is set 2021-05-12T12:56:23.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:153: TEST_recovery_priority: ceph osd pool set test7 size 2 2021-05-12T12:56:24.335 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 7 size to 2 2021-05-12T12:56:24.344 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:154: TEST_recovery_priority: sleep 2 2021-05-12T12:56:26.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:56:26.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:56:26.348 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:56:26.348 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:56:26.348 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:56:26.348 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:56:26.349 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:56:26.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:56:26.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:56:26.451 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:56:26.451 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:56:26.451 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:26.452 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:26.452 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:26.452 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:56:26.452 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:26.453 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:56:26.453 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:26.453 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:26.453 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:26.453 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:56:26.454 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:26.454 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:56:26.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: seq 1 10 2021-05-12T12:56:26.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-05-12T12:56:26.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-05-12T12:56:26.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-05-12T12:56:26.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 1 = 10 ']' 2021-05-12T12:56:26.837 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2021-05-12T12:56:28.838 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-05-12T12:56:28.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-05-12T12:56:28.839 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-05-12T12:56:29.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 2 = 10 ']' 2021-05-12T12:56:29.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2021-05-12T12:56:31.201 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-05-12T12:56:31.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-05-12T12:56:31.202 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-05-12T12:56:31.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: break 2021-05-12T12:56:31.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:169: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:56:31.596 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:56:31.596 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:56:31.956 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:56:31.957 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:56:31.957 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:56:31.957 INFO:tasks.workunit.client.0.smithi191.stderr:3 2021-05-12T12:56:31.957 INFO:tasks.workunit.client.0.smithi191.stderr:4' 2021-05-12T12:56:31.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:56:31.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:31.958 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:56:32.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836577 2021-05-12T12:56:32.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836577 2021-05-12T12:56:32.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836577' 2021-05-12T12:56:32.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:32.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:56:32.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705760 2021-05-12T12:56:32.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705760 2021-05-12T12:56:32.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836577 1-38654705760' 2021-05-12T12:56:32.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:32.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:56:32.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542238 2021-05-12T12:56:32.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542238 2021-05-12T12:56:32.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836577 1-38654705760 2-60129542238' 2021-05-12T12:56:32.339 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:32.339 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:56:32.465 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=81604378716 2021-05-12T12:56:32.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 81604378716 2021-05-12T12:56:32.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836577 1-38654705760 2-60129542238 3-81604378716' 2021-05-12T12:56:32.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:32.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:56:32.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=103079215194 2021-05-12T12:56:32.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 103079215194 2021-05-12T12:56:32.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836577 1-38654705760 2-60129542238 3-81604378716 4-103079215194' 2021-05-12T12:56:32.593 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:32.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836577 2021-05-12T12:56:32.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:32.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:56:32.594 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836577 2021-05-12T12:56:32.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:32.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836577 2021-05-12T12:56:32.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2021-05-12T12:56:32.595 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836577 2021-05-12T12:56:32.596 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:56:32.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836575 -lt 21474836577 2021-05-12T12:56:32.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:56:33.952 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:56:33.953 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:56:34.321 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836578 -lt 21474836577 2021-05-12T12:56:34.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:34.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705760 2021-05-12T12:56:34.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:34.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:56:34.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705760 2021-05-12T12:56:34.324 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:34.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705760 2021-05-12T12:56:34.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705760' 2021-05-12T12:56:34.325 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 38654705760 2021-05-12T12:56:34.326 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:56:34.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705761 -lt 38654705760 2021-05-12T12:56:34.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:34.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542238 2021-05-12T12:56:34.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:34.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:56:34.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542238 2021-05-12T12:56:34.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:34.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542238 2021-05-12T12:56:34.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542238' 2021-05-12T12:56:34.706 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542238 2021-05-12T12:56:34.707 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:56:35.089 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542238 -lt 60129542238 2021-05-12T12:56:35.090 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:35.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378716 2021-05-12T12:56:35.090 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:35.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:56:35.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-81604378716 2021-05-12T12:56:35.091 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:35.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=81604378716 2021-05-12T12:56:35.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 81604378716' 2021-05-12T12:56:35.093 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 81604378716 2021-05-12T12:56:35.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:56:35.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 81604378716 -lt 81604378716 2021-05-12T12:56:35.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:35.507 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215194 2021-05-12T12:56:35.507 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:35.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:56:35.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-103079215194 2021-05-12T12:56:35.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:35.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=103079215194 2021-05-12T12:56:35.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 103079215194' 2021-05-12T12:56:35.509 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.4 seq 103079215194 2021-05-12T12:56:35.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:56:35.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 103079215194 -lt 103079215194 2021-05-12T12:56:35.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:56:35.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:56:35.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:56:35.895 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:56:35.896 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:56:35.896 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:56:35.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:56:35.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:56:35.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:56:35.987 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:56:35.988 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:56:35.988 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:35.988 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:35.988 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:35.988 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:56:35.989 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:35.989 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:56:35.989 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:56:35.989 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:35.990 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:35.990 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:56:35.990 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:35.990 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:56:35.990 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:35.991 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:35.991 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:35.991 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:56:35.991 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:35.992 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:56:35.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: ceph osd out osd.0 2021-05-12T12:56:36.491 INFO:tasks.workunit.client.0.smithi191.stderr:osd.0 is already out. 2021-05-12T12:56:36.500 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: sleep 2 2021-05-12T12:56:38.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:174: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:56:38.502 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:56:38.503 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:56:38.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:56:38.872 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:56:38.872 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:56:38.873 INFO:tasks.workunit.client.0.smithi191.stderr:3 2021-05-12T12:56:38.873 INFO:tasks.workunit.client.0.smithi191.stderr:4' 2021-05-12T12:56:38.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:56:38.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:38.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:56:38.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836581 2021-05-12T12:56:38.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836581 2021-05-12T12:56:39.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836581' 2021-05-12T12:56:39.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:39.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:56:39.127 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705764 2021-05-12T12:56:39.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705764 2021-05-12T12:56:39.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836581 1-38654705764' 2021-05-12T12:56:39.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:39.129 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:56:39.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542241 2021-05-12T12:56:39.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542241 2021-05-12T12:56:39.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836581 1-38654705764 2-60129542241' 2021-05-12T12:56:39.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:39.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:56:39.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=81604378719 2021-05-12T12:56:39.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 81604378719 2021-05-12T12:56:39.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836581 1-38654705764 2-60129542241 3-81604378719' 2021-05-12T12:56:39.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:39.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:56:39.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=103079215197 2021-05-12T12:56:39.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 103079215197 2021-05-12T12:56:39.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836581 1-38654705764 2-60129542241 3-81604378719 4-103079215197' 2021-05-12T12:56:39.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:39.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836581 2021-05-12T12:56:39.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:39.511 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:56:39.512 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836581 2021-05-12T12:56:39.512 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:39.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836581 2021-05-12T12:56:39.513 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2021-05-12T12:56:39.513 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836581 2021-05-12T12:56:39.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:56:39.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836581 -lt 21474836581 2021-05-12T12:56:39.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:39.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705764 2021-05-12T12:56:39.871 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:39.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:56:39.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705764 2021-05-12T12:56:39.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:39.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705764 2021-05-12T12:56:39.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705764' 2021-05-12T12:56:39.873 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 38654705764 2021-05-12T12:56:39.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:56:40.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705764 -lt 38654705764 2021-05-12T12:56:40.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:40.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542241 2021-05-12T12:56:40.236 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:40.236 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:56:40.237 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542241 2021-05-12T12:56:40.237 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:40.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542241 2021-05-12T12:56:40.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542241' 2021-05-12T12:56:40.238 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542241 2021-05-12T12:56:40.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:56:40.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542241 -lt 60129542241 2021-05-12T12:56:40.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:40.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378719 2021-05-12T12:56:40.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:40.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:56:40.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-81604378719 2021-05-12T12:56:40.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:40.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=81604378719 2021-05-12T12:56:40.694 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 81604378719' 2021-05-12T12:56:40.694 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 81604378719 2021-05-12T12:56:40.695 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:56:41.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 81604378717 -lt 81604378719 2021-05-12T12:56:41.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:56:42.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:56:42.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:56:42.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 81604378719 -lt 81604378719 2021-05-12T12:56:42.440 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:42.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215197 2021-05-12T12:56:42.441 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:42.441 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:56:42.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-103079215197 2021-05-12T12:56:42.442 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:42.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=103079215197 2021-05-12T12:56:42.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 103079215197' 2021-05-12T12:56:42.444 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.4 seq 103079215197 2021-05-12T12:56:42.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:56:42.831 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 103079215197 -lt 103079215197 2021-05-12T12:56:42.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:56:42.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:56:42.833 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:56:42.834 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:56:42.834 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:56:42.835 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:56:42.835 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:56:42.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:56:42.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:56:42.924 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:56:42.925 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:56:42.925 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:42.925 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:42.926 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:42.926 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:56:42.926 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:42.926 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:56:42.927 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:56:42.927 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:42.927 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:42.927 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:42.927 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:56:42.928 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:56:42.928 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:42.928 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:42.928 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:56:42.929 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:42.929 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:56:42.929 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:42.929 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:42.929 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:42.930 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:56:42.930 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:42.930 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:56:42.933 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: ceph pg dump pgs 2021-05-12T12:56:43.267 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:56:43.267 INFO:tasks.workunit.client.0.smithi191.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2021-05-12T12:56:31.485435+0000 93'600 104:670 [1,4] 1 [1] 1 0'0 2021-05-12T12:49:52.968683+0000 0'0 2021-05-12T12:49:52.968683+0000 0 2021-05-12T12:56:43.268 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 200 0 0 200 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T12:56:37.849904+0000 93'600 105:687 [1,2] 1 [1] 1 0'0 2021-05-12T12:49:38.387886+0000 0'0 2021-05-12T12:49:38.387886+0000 0 2021-05-12T12:56:43.268 INFO:tasks.workunit.client.0.smithi191.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:50:22.967081+0000 93'600 104:670 [1] 1 [1] 1 0'0 2021-05-12T12:49:43.135094+0000 0'0 2021-05-12T12:49:43.135094+0000 0 2021-05-12T12:56:43.269 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:56:43.269 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:56:43.269 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:56:43.278 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph osd pool set test3 size 2 2021-05-12T12:56:43.935 INFO:tasks.workunit.client.0.smithi191.stderr:set pool 3 size to 2 2021-05-12T12:56:43.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: sleep 2 2021-05-12T12:56:45.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:180: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:56:45.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:56:45.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:56:46.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:56:46.324 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:56:46.324 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:56:46.324 INFO:tasks.workunit.client.0.smithi191.stderr:3 2021-05-12T12:56:46.325 INFO:tasks.workunit.client.0.smithi191.stderr:4' 2021-05-12T12:56:46.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:56:46.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:46.325 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:56:46.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836584 2021-05-12T12:56:46.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836584 2021-05-12T12:56:46.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584' 2021-05-12T12:56:46.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:46.471 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:56:46.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705767 2021-05-12T12:56:46.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705767 2021-05-12T12:56:46.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584 1-38654705767' 2021-05-12T12:56:46.598 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:46.599 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:56:46.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542244 2021-05-12T12:56:46.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542244 2021-05-12T12:56:46.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584 1-38654705767 2-60129542244' 2021-05-12T12:56:46.724 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:46.725 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:56:46.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=81604378723 2021-05-12T12:56:46.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 81604378723 2021-05-12T12:56:46.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584 1-38654705767 2-60129542244 3-81604378723' 2021-05-12T12:56:46.851 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:46.851 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:56:46.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=103079215200 2021-05-12T12:56:46.976 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 103079215200 2021-05-12T12:56:46.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584 1-38654705767 2-60129542244 3-81604378723 4-103079215200' 2021-05-12T12:56:46.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:46.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836584 2021-05-12T12:56:46.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:46.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:56:46.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836584 2021-05-12T12:56:46.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:46.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836584 2021-05-12T12:56:46.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2021-05-12T12:56:46.980 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836584 2021-05-12T12:56:46.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:56:47.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836582 -lt 21474836584 2021-05-12T12:56:47.337 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:56:48.338 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:56:48.339 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:56:48.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836584 -lt 21474836584 2021-05-12T12:56:48.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:48.704 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705767 2021-05-12T12:56:48.704 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:48.704 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:56:48.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705767 2021-05-12T12:56:48.705 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:48.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705767 2021-05-12T12:56:48.706 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705767' 2021-05-12T12:56:48.706 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 38654705767 2021-05-12T12:56:48.707 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:56:49.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705767 -lt 38654705767 2021-05-12T12:56:49.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:49.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542244 2021-05-12T12:56:49.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:49.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:56:49.086 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542244 2021-05-12T12:56:49.087 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:49.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542244 2021-05-12T12:56:49.087 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542244' 2021-05-12T12:56:49.088 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542244 2021-05-12T12:56:49.088 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:56:49.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542245 -lt 60129542244 2021-05-12T12:56:49.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:49.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378723 2021-05-12T12:56:49.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:49.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:56:49.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-81604378723 2021-05-12T12:56:49.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:49.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=81604378723 2021-05-12T12:56:49.471 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 81604378723' 2021-05-12T12:56:49.471 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 81604378723 2021-05-12T12:56:49.472 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:56:49.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 81604378723 -lt 81604378723 2021-05-12T12:56:49.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:49.860 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215200 2021-05-12T12:56:49.860 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:49.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:56:49.861 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-103079215200 2021-05-12T12:56:49.861 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:49.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=103079215200 2021-05-12T12:56:49.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 103079215200' 2021-05-12T12:56:49.863 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.4 seq 103079215200 2021-05-12T12:56:49.863 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:56:50.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 103079215201 -lt 103079215200 2021-05-12T12:56:50.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:56:50.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:56:50.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:56:50.220 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:56:50.221 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:56:50.221 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:56:50.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:56:50.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:56:50.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:56:50.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:56:50.320 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:56:50.320 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:56:50.320 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:50.320 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:50.321 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:50.321 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:56:50.321 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:50.321 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:56:50.322 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:56:50.322 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:50.322 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:50.322 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:50.322 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:56:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:56:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:50.323 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:56:50.324 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:56:50.324 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:50.324 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:50.324 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:56:50.324 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:50.325 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:56:50.325 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:50.325 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:50.325 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:50.325 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:56:50.326 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:50.326 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:56:50.326 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: ceph pg dump pgs 2021-05-12T12:56:50.660 INFO:tasks.workunit.client.0.smithi191.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:56:50.660 INFO:tasks.workunit.client.0.smithi191.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2021-05-12T12:56:31.485435+0000 93'600 108:673 [1,4] 1 [1] 1 0'0 2021-05-12T12:49:52.968683+0000 0'0 2021-05-12T12:49:52.968683+0000 0 2021-05-12T12:56:50.660 INFO:tasks.workunit.client.0.smithi191.stdout:1.0 200 0 0 200 0 2097152000 0 0 600 600 active+recovering+undersized+remapped 2021-05-12T12:56:44.223279+0000 93'600 108:690 [1,2] 1 [1] 1 0'0 2021-05-12T12:49:38.387886+0000 0'0 2021-05-12T12:49:38.387886+0000 0 2021-05-12T12:56:50.660 INFO:tasks.workunit.client.0.smithi191.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T12:56:45.027541+0000 93'600 109:681 [1,2] 1 [1] 1 0'0 2021-05-12T12:49:43.135094+0000 0'0 2021-05-12T12:49:43.135094+0000 0 2021-05-12T12:56:50.661 INFO:tasks.workunit.client.0.smithi191.stdout: 2021-05-12T12:56:50.661 INFO:tasks.workunit.client.0.smithi191.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:56:50.662 INFO:tasks.workunit.client.0.smithi191.stderr:dumped pgs 2021-05-12T12:56:50.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:56:50.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2021-05-12T12:56:50.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: PRIO= 2021-05-12T12:56:50.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: '[' '' '!=' 190 ']' 2021-05-12T12:56:50.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: echo 'The normal PG 1.0 doesn'\''t have prio 190 queued waiting' 2021-05-12T12:56:50.675 INFO:tasks.workunit.client.0.smithi191.stdout:The normal PG 1.0 doesn't have prio 190 queued waiting 2021-05-12T12:56:50.676 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: expr 0 + 1 2021-05-12T12:56:50.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: ERRORS=1 2021-05-12T12:56:50.676 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:56:50.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:56:50.678 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: eval 'ITEM="1.0"' 2021-05-12T12:56:50.679 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: ITEM=1.0 2021-05-12T12:56:50.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:194: TEST_recovery_priority: '[' 1.0 '!=' 7.0 ']' 2021-05-12T12:56:50.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: echo 'The first force-recovery PG 7.0 didn'\''t become the in progress item' 2021-05-12T12:56:50.680 INFO:tasks.workunit.client.0.smithi191.stdout:The first force-recovery PG 7.0 didn't become the in progress item 2021-05-12T12:56:50.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: expr 1 + 1 2021-05-12T12:56:50.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: ERRORS=2 2021-05-12T12:56:50.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: seq 1 10 2021-05-12T12:56:50.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-05-12T12:56:50.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: ceph pg force-recovery 3.0 2021-05-12T12:56:50.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-05-12T12:56:51.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: break 2021-05-12T12:56:51.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:219: TEST_recovery_priority: sleep 2 2021-05-12T12:56:53.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:56:53.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:56:53.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:56:53.041 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:56:53.041 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:56:53.041 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:56:53.041 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:56:53.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:56:53.042 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:56:53.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:221: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:56:53.154 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:56:53.155 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:56:53.155 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:53.155 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:53.155 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:53.155 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:56:53.156 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:53.156 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:56:53.156 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:56:53.156 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:53.157 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:53.157 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:53.157 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:56:53.157 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:56:53.157 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:53.158 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:53.158 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:53.158 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:56:53.158 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:56:53.158 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:53.159 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:53.159 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:56:53.159 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:53.159 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:56:53.159 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:53.160 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:53.160 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:53.160 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:56:53.160 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:53.160 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:56:53.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:56:53.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-05-12T12:56:53.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: PRIO= 2021-05-12T12:56:53.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: '[' '' '!=' 255 ']' 2021-05-12T12:56:53.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: echo 'The second force-recovery PG 3.0 doesn'\''t have prio 255' 2021-05-12T12:56:53.163 INFO:tasks.workunit.client.0.smithi191.stdout:The second force-recovery PG 3.0 doesn't have prio 255 2021-05-12T12:56:53.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: expr 2 + 1 2021-05-12T12:56:53.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: ERRORS=3 2021-05-12T12:56:53.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:228: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:56:53.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:56:53.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:56:53.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:56:53.524 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:56:53.524 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:56:53.525 INFO:tasks.workunit.client.0.smithi191.stderr:3 2021-05-12T12:56:53.525 INFO:tasks.workunit.client.0.smithi191.stderr:4' 2021-05-12T12:56:53.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:56:53.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:53.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:56:53.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836588 2021-05-12T12:56:53.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836588 2021-05-12T12:56:53.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836588' 2021-05-12T12:56:53.651 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:53.652 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:56:53.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705771 2021-05-12T12:56:53.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705771 2021-05-12T12:56:53.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836588 1-38654705771' 2021-05-12T12:56:53.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:53.779 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:56:53.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542248 2021-05-12T12:56:53.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542248 2021-05-12T12:56:53.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836588 1-38654705771 2-60129542248' 2021-05-12T12:56:53.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:53.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:56:54.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=81604378726 2021-05-12T12:56:54.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 81604378726 2021-05-12T12:56:54.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836588 1-38654705771 2-60129542248 3-81604378726' 2021-05-12T12:56:54.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:56:54.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:56:54.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=103079215204 2021-05-12T12:56:54.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 103079215204 2021-05-12T12:56:54.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836588 1-38654705771 2-60129542248 3-81604378726 4-103079215204' 2021-05-12T12:56:54.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:54.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836588 2021-05-12T12:56:54.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:54.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:56:54.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836588 2021-05-12T12:56:54.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:54.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836588 2021-05-12T12:56:54.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2021-05-12T12:56:54.165 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836588 2021-05-12T12:56:54.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:56:54.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836586 -lt 21474836588 2021-05-12T12:56:54.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:56:55.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:56:55.522 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:56:55.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836588 -lt 21474836588 2021-05-12T12:56:55.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:55.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705771 2021-05-12T12:56:55.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:55.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:56:55.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705771 2021-05-12T12:56:55.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:55.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705771 2021-05-12T12:56:55.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705771' 2021-05-12T12:56:55.909 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 38654705771 2021-05-12T12:56:55.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:56:56.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705771 -lt 38654705771 2021-05-12T12:56:56.272 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:56.273 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542248 2021-05-12T12:56:56.273 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:56.273 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:56:56.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542248 2021-05-12T12:56:56.274 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:56.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542248 2021-05-12T12:56:56.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542248' 2021-05-12T12:56:56.276 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542248 2021-05-12T12:56:56.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:56:56.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542248 -lt 60129542248 2021-05-12T12:56:56.638 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:56.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378726 2021-05-12T12:56:56.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:56.639 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:56:56.639 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-81604378726 2021-05-12T12:56:56.640 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:56.640 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=81604378726 2021-05-12T12:56:56.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 81604378726' 2021-05-12T12:56:56.641 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 81604378726 2021-05-12T12:56:56.641 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:56:56.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 81604378726 -lt 81604378726 2021-05-12T12:56:56.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:56:56.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215204 2021-05-12T12:56:56.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:56:56.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:56:56.999 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-103079215204 2021-05-12T12:56:57.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:56:57.000 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=103079215204 2021-05-12T12:56:57.001 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 103079215204' 2021-05-12T12:56:57.001 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.4 seq 103079215204 2021-05-12T12:56:57.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:56:57.390 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 103079215204 -lt 103079215204 2021-05-12T12:56:57.391 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:232: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2021-05-12T12:56:57.733 INFO:tasks.workunit.client.0.smithi191.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2021-05-12T12:56:57.734 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T12:56:57.742 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:233: TEST_recovery_priority: sleep 2 2021-05-12T12:56:59.744 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:56:59.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:56:59.745 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:56:59.745 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:56:59.745 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:56:59.746 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:56:59.746 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:56:59.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:56:59.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:56:59.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:56:59.858 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:56:59.858 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:56:59.858 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:59.858 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:59.859 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:59.859 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:56:59.859 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:59.859 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:56:59.860 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:56:59.860 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:59.860 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:59.860 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:59.860 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:56:59.861 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:56:59.861 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:59.861 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:59.861 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:56:59.861 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:56:59.862 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:56:59.862 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:56:59.862 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:59.862 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:56:59.862 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:56:59.863 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:56:59.863 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:56:59.863 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:56:59.863 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:56:59.864 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:56:59.864 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:56:59.864 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:56:59.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:56:59.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2021-05-12T12:56:59.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: PRIO= 2021-05-12T12:56:59.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: '[' '' '!=' 190 ']' 2021-05-12T12:56:59.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: echo 'After cancel-recovery PG 7.0 doesn'\''t have prio 190' 2021-05-12T12:56:59.867 INFO:tasks.workunit.client.0.smithi191.stdout:After cancel-recovery PG 7.0 doesn't have prio 190 2021-05-12T12:56:59.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: expr 3 + 1 2021-05-12T12:56:59.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: ERRORS=4 2021-05-12T12:56:59.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:56:59.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:56:59.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: eval 'ITEM="1.0"' 2021-05-12T12:56:59.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: ITEM=1.0 2021-05-12T12:56:59.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:245: TEST_recovery_priority: '[' 1.0 '!=' 3.0 ']' 2021-05-12T12:56:59.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: echo 'The force-recovery PG 3.0 didn'\''t become the in progress item' 2021-05-12T12:56:59.870 INFO:tasks.workunit.client.0.smithi191.stdout:The force-recovery PG 3.0 didn't become the in progress item 2021-05-12T12:56:59.871 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: expr 4 + 1 2021-05-12T12:56:59.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: ERRORS=5 2021-05-12T12:56:59.871 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:258: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2021-05-12T12:57:00.209 INFO:tasks.workunit.client.0.smithi191.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2021-05-12T12:57:00.210 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T12:57:00.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:259: TEST_recovery_priority: sleep 5 2021-05-12T12:57:05.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:57:05.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:57:05.222 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:57:05.222 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:57:05.222 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:57:05.222 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:57:05.223 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:57:05.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:57:05.223 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:05.324 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:05.324 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:05.324 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:05.325 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:05.325 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:05.325 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:05.325 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:05.325 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:05.326 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:05.326 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:05.326 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:05.326 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:05.327 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:05.327 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:05.327 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:05.327 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:05.327 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:05.328 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:05.328 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:05.328 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:05.328 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:05.329 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:05.329 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:05.329 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:05.329 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:05.329 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:05.330 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:05.330 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:05.330 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:05.330 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:05.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:57:05.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:57:05.333 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:57:05.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:57:05.692 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:57:05.692 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:57:05.692 INFO:tasks.workunit.client.0.smithi191.stderr:3 2021-05-12T12:57:05.692 INFO:tasks.workunit.client.0.smithi191.stderr:4' 2021-05-12T12:57:05.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:57:05.693 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:05.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:57:05.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836592 2021-05-12T12:57:05.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836592 2021-05-12T12:57:05.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836592' 2021-05-12T12:57:05.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:05.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:57:05.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705775 2021-05-12T12:57:05.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705775 2021-05-12T12:57:05.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836592 1-38654705775' 2021-05-12T12:57:05.972 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:05.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:57:06.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542252 2021-05-12T12:57:06.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542252 2021-05-12T12:57:06.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836592 1-38654705775 2-60129542252' 2021-05-12T12:57:06.099 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:06.099 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:57:06.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=81604378730 2021-05-12T12:57:06.226 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 81604378730 2021-05-12T12:57:06.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836592 1-38654705775 2-60129542252 3-81604378730' 2021-05-12T12:57:06.227 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:06.227 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:57:06.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=103079215208 2021-05-12T12:57:06.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 103079215208 2021-05-12T12:57:06.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836592 1-38654705775 2-60129542252 3-81604378730 4-103079215208' 2021-05-12T12:57:06.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:06.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836592 2021-05-12T12:57:06.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:06.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:57:06.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836592 2021-05-12T12:57:06.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:06.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836592 2021-05-12T12:57:06.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2021-05-12T12:57:06.357 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836592 2021-05-12T12:57:06.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:06.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836590 -lt 21474836592 2021-05-12T12:57:06.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:57:07.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:57:07.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:08.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836592 -lt 21474836592 2021-05-12T12:57:08.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:08.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705775 2021-05-12T12:57:08.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:08.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:57:08.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705775 2021-05-12T12:57:08.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:08.085 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705775 2021-05-12T12:57:08.086 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705775' 2021-05-12T12:57:08.086 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 38654705775 2021-05-12T12:57:08.086 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:57:08.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705775 -lt 38654705775 2021-05-12T12:57:08.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:08.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542252 2021-05-12T12:57:08.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:08.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:57:08.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542252 2021-05-12T12:57:08.482 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:08.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542252 2021-05-12T12:57:08.483 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542252 2021-05-12T12:57:08.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542252' 2021-05-12T12:57:08.484 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:57:08.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542253 -lt 60129542252 2021-05-12T12:57:08.869 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:08.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378730 2021-05-12T12:57:08.870 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:08.870 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:57:08.871 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-81604378730 2021-05-12T12:57:08.872 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:08.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=81604378730 2021-05-12T12:57:08.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 81604378730' 2021-05-12T12:57:08.872 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 81604378730 2021-05-12T12:57:08.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:57:09.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 81604378731 -lt 81604378730 2021-05-12T12:57:09.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:09.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215208 2021-05-12T12:57:09.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:09.250 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:57:09.250 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-103079215208 2021-05-12T12:57:09.251 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:09.251 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=103079215208 2021-05-12T12:57:09.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 103079215208' 2021-05-12T12:57:09.252 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.4 seq 103079215208 2021-05-12T12:57:09.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:57:09.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 103079215209 -lt 103079215208 2021-05-12T12:57:09.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:264: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-05-12T12:57:09.979 INFO:tasks.workunit.client.0.smithi191.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2021-05-12T12:57:09.979 INFO:tasks.workunit.client.0.smithi191.stderr: 2021-05-12T12:57:09.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:265: TEST_recovery_priority: sleep 2 2021-05-12T12:57:11.991 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:57:11.991 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:57:11.991 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:57:11.991 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:57:11.992 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:57:11.992 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:57:11.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:57:11.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:57:11.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:12.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:12.103 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:12.103 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:12.104 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:12.104 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:12.104 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:12.104 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:12.104 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:12.105 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:12.105 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:12.105 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:12.105 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:12.106 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:12.106 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:12.106 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:12.106 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:12.107 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:12.107 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:12.107 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:12.107 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:12.107 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:12.108 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:12.108 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:12.108 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:12.108 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:12.109 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:12.109 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:12.109 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:12.109 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:12.109 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:12.110 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:12.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:12.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-05-12T12:57:12.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: PRIO= 2021-05-12T12:57:12.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: '[' '' '!=' 190 ']' 2021-05-12T12:57:12.112 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: echo 'After cancel-force-recovery PG 7.0 doesn'\''t have prio 190' 2021-05-12T12:57:12.112 INFO:tasks.workunit.client.0.smithi191.stdout:After cancel-force-recovery PG 7.0 doesn't have prio 190 2021-05-12T12:57:12.113 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: expr 5 + 1 2021-05-12T12:57:12.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: ERRORS=6 2021-05-12T12:57:12.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:12.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:57:12.114 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: eval 'ITEM="1.0"' 2021-05-12T12:57:12.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: ITEM=1.0 2021-05-12T12:57:12.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:277: TEST_recovery_priority: '[' 1.0 '!=' 7.0 ']' 2021-05-12T12:57:12.115 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: echo 'The force-recovery PG 7.0 didn'\''t get promoted to an in progress item' 2021-05-12T12:57:12.115 INFO:tasks.workunit.client.0.smithi191.stdout:The force-recovery PG 7.0 didn't get promoted to an in progress item 2021-05-12T12:57:12.116 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: expr 6 + 1 2021-05-12T12:57:12.116 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: ERRORS=7 2021-05-12T12:57:12.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:290: TEST_recovery_priority: ceph osd unset noout 2021-05-12T12:57:12.782 INFO:tasks.workunit.client.0.smithi191.stderr:noout is unset 2021-05-12T12:57:12.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:291: TEST_recovery_priority: ceph osd unset norecover 2021-05-12T12:57:13.575 INFO:tasks.workunit.client.0.smithi191.stderr:norecover is unset 2021-05-12T12:57:13.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:57:13.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:57:13.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:57:13.588 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:57:13.588 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:57:13.588 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.190757 2021-05-12T12:57:13.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.190757/ceph-osd.1.asok 2021-05-12T12:57:13.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations' 2021-05-12T12:57:13.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations' 2021-05-12T12:57:13.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:57:13.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:57:13.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:57:13.590 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:57:13.590 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:57:13.590 INFO:tasks.workunit.client.0.smithi191.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:57:13.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:57:13.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:57:13.591 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:57:13.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:57:13.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:57:13.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:57:13.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:57:13.676 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:57:14.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:57:14.036 INFO:tasks.workunit.client.0.smithi191.stderr:1 2021-05-12T12:57:14.037 INFO:tasks.workunit.client.0.smithi191.stderr:2 2021-05-12T12:57:14.037 INFO:tasks.workunit.client.0.smithi191.stderr:3 2021-05-12T12:57:14.037 INFO:tasks.workunit.client.0.smithi191.stderr:4' 2021-05-12T12:57:14.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:57:14.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:14.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:57:14.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836596 2021-05-12T12:57:14.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836596 2021-05-12T12:57:14.163 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836596' 2021-05-12T12:57:14.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:14.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:57:14.290 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705779 2021-05-12T12:57:14.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705779 2021-05-12T12:57:14.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836596 1-38654705779' 2021-05-12T12:57:14.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:14.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:57:14.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542256 2021-05-12T12:57:14.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542256 2021-05-12T12:57:14.418 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836596 1-38654705779 2-60129542256' 2021-05-12T12:57:14.419 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:14.419 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:57:14.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=81604378734 2021-05-12T12:57:14.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 81604378734 2021-05-12T12:57:14.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836596 1-38654705779 2-60129542256 3-81604378734' 2021-05-12T12:57:14.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:14.548 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:57:14.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=103079215212 2021-05-12T12:57:14.679 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 103079215212 2021-05-12T12:57:14.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836596 1-38654705779 2-60129542256 3-81604378734 4-103079215212' 2021-05-12T12:57:14.680 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:14.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836596 2021-05-12T12:57:14.680 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:14.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:57:14.681 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836596 2021-05-12T12:57:14.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:14.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836596 2021-05-12T12:57:14.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2021-05-12T12:57:14.683 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.0 seq 21474836596 2021-05-12T12:57:14.683 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:15.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836594 -lt 21474836596 2021-05-12T12:57:15.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:57:16.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:57:16.056 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:16.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836596 -lt 21474836596 2021-05-12T12:57:16.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:16.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705779 2021-05-12T12:57:16.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:16.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:57:16.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705779 2021-05-12T12:57:16.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:16.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705779 2021-05-12T12:57:16.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705779' 2021-05-12T12:57:16.461 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.1 seq 38654705779 2021-05-12T12:57:16.461 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:57:16.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705779 -lt 38654705779 2021-05-12T12:57:16.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:16.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542256 2021-05-12T12:57:16.866 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:16.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:57:16.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542256 2021-05-12T12:57:16.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:16.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542256 2021-05-12T12:57:16.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542256' 2021-05-12T12:57:16.868 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.2 seq 60129542256 2021-05-12T12:57:16.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:57:17.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542256 -lt 60129542256 2021-05-12T12:57:17.238 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:17.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378734 2021-05-12T12:57:17.239 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:17.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:57:17.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-81604378734 2021-05-12T12:57:17.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:17.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=81604378734 2021-05-12T12:57:17.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 81604378734' 2021-05-12T12:57:17.242 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.3 seq 81604378734 2021-05-12T12:57:17.242 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:57:17.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 81604378735 -lt 81604378734 2021-05-12T12:57:17.634 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:17.635 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215212 2021-05-12T12:57:17.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:17.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:57:17.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-103079215212 2021-05-12T12:57:17.636 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:17.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=103079215212 2021-05-12T12:57:17.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 103079215212' 2021-05-12T12:57:17.638 INFO:tasks.workunit.client.0.smithi191.stdout:waiting osd.4 seq 103079215212 2021-05-12T12:57:17.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:57:18.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 103079215212 -lt 103079215212 2021-05-12T12:57:18.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:57:18.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:18.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:18.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 3 == 0 2021-05-12T12:57:18.507 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:18.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:18.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:18.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:18.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:18.508 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:18.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:18.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:18.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:18.887 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:18.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:19.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:19.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:57:19.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:57:19.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:57:19.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:19.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:19.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:19.467 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:19.468 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:19.468 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:19.468 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:19.468 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:19.469 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:19.469 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:19.469 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:19.469 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:19.469 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:19.470 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:19.470 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:19.470 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:19.470 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:19.471 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:19.471 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:19.471 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:19.471 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:19.472 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:19.472 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:19.472 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:19.472 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:19.472 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:19.473 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:19.473 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:19.473 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:19.473 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:19.473 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:19.474 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:19.474 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:19.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:19.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:19.577 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:19.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:19.578 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:19.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:19.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:19.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:19.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:19.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:19.936 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:19.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:19.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:20.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:20.420 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:20.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:20.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:20.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:20.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:20.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:20.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:20.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:20.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3236778 2021-05-12T12:57:20.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3236778 '!=' null 2021-05-12T12:57:20.925 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:20.926 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:20.926 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:20.926 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:21.019 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:21.020 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:21.020 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:21.020 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:21.020 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:21.021 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:21.021 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:21.021 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:21.021 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:21.022 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:21.022 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:21.022 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:21.023 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:21.023 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:21.023 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:21.023 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:21.023 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:21.024 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:21.024 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:21.024 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:21.024 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:21.024 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:21.025 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:21.025 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:21.025 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:21.025 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:21.026 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:21.026 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:21.026 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:21.026 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:21.028 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:21.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:21.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:21.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:21.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:21.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:21.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:21.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:21.131 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:21.492 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:21.492 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:21.493 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:21.493 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:21.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:21.986 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:21.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:21.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:21.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:21.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:21.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:21.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:21.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:22.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620966 2021-05-12T12:57:22.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620966 '!=' null 2021-05-12T12:57:22.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:22.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:22.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:22.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:22.620 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:22.620 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:22.621 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:22.621 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:22.621 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:22.621 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:22.621 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:22.622 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:22.622 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:22.622 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:22.622 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:22.623 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:22.623 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:22.623 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:22.623 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:22.623 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:22.624 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:22.624 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:22.624 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:22.624 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:22.625 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:22.625 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:22.625 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:22.625 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:22.625 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:22.626 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:22.626 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:22.626 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:22.626 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:22.627 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:22.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:22.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:22.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:22.731 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:22.731 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:22.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:22.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:22.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:22.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:23.084 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:23.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:23.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:23.086 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:23.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:23.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:23.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:23.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:23.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:23.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:23.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:23.573 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:23.573 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:24.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5245152 2021-05-12T12:57:24.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5245152 '!=' null 2021-05-12T12:57:24.054 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:24.055 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:24.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:24.055 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:24.148 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:24.148 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:24.148 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:24.149 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:24.149 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:24.149 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:24.149 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:24.149 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:24.150 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:24.150 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:24.150 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:24.150 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:24.150 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:24.151 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:24.151 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:24.151 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:24.151 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:24.151 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:24.152 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:24.152 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:24.152 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:24.152 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:24.152 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:24.153 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:24.153 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:24.153 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:24.153 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:24.153 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:24.154 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:24.154 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:24.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:24.258 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:24.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:24.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:24.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:24.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:24.259 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:24.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:24.260 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:24.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:24.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:24.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:24.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:25.107 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:25.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:25.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:25.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:25.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:25.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:25.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:25.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:25.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:25.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4486275 2021-05-12T12:57:25.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4486275 '!=' null 2021-05-12T12:57:25.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:25.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:25.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:25.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:25.688 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:25.689 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:25.689 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:25.689 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:25.689 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:25.689 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:25.690 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:25.690 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:25.690 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:25.690 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:25.690 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:25.691 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:25.691 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:25.691 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:25.691 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:25.691 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:25.692 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:25.692 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:25.692 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:25.692 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:25.692 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:25.693 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:25.693 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:25.693 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:25.693 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:25.694 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:25.694 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:25.694 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:25.694 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:25.694 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:25.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:25.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:25.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:25.799 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:25.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:25.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:25.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:25.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:25.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:26.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:26.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:26.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:26.165 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:26.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:26.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:26.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:26.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:26.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:26.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:26.654 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:26.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:26.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:27.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4486275 2021-05-12T12:57:27.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4486275 '!=' null 2021-05-12T12:57:27.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:27.194 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:27.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:27.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:27.289 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:27.289 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:27.289 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:27.290 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:27.290 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:27.290 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:27.290 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:27.290 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:27.291 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:27.291 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:27.291 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:27.291 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:27.292 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:27.292 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:27.292 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:27.292 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:27.292 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:27.293 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:27.293 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:27.293 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:27.293 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:27.294 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:27.294 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:27.294 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:27.294 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:27.294 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:27.295 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:27.295 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:27.295 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:27.295 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:27.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:27.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:27.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:27.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:27.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:27.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:27.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:27.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:27.401 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:27.754 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:27.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:27.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:27.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:28.243 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:28.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:28.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:28.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:28.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:28.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:28.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:28.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:28.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:28.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368253 2021-05-12T12:57:28.775 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368253 '!=' null 2021-05-12T12:57:28.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:28.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:28.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:28.776 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:28.868 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:28.868 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:28.869 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:28.869 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:28.869 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:28.869 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:28.869 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:28.870 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:28.870 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:28.870 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:28.870 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:28.871 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:28.871 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:28.871 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:28.871 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:28.871 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:28.872 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:28.872 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:28.872 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:28.872 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:28.872 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:28.873 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:28.873 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:28.873 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:28.873 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:28.874 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:28.874 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:28.874 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:28.874 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:28.874 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:28.876 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:28.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:28.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:28.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:28.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:28.979 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:28.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:28.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:28.980 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:29.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:29.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:29.335 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:29.335 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:29.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:29.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:29.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:29.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:29.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:29.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:29.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:29.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:29.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:30.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115536 2021-05-12T12:57:30.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115536 '!=' null 2021-05-12T12:57:30.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:30.312 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:30.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:30.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:30.418 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:30.418 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:30.419 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:30.419 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:30.419 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:30.419 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:30.420 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:30.420 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:30.420 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:30.420 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:30.421 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:30.421 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:30.421 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:30.421 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:30.421 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:30.422 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:30.422 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:30.422 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:30.422 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:30.422 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:30.423 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:30.423 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:30.423 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:30.423 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:30.424 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:30.424 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:30.424 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:30.424 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:30.424 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:30.425 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:30.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:30.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:30.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:30.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:30.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:30.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:30.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:30.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:30.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:30.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:30.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:30.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:30.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:31.380 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:31.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:31.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:31.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:31.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:31.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:31.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:31.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:31.382 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:31.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368269 2021-05-12T12:57:31.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368269 '!=' null 2021-05-12T12:57:31.864 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:31.865 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:31.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:31.865 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:31.957 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:31.957 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:31.957 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:31.958 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:31.958 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:31.958 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:31.958 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:31.959 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:31.959 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:31.959 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:31.959 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:31.959 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:31.960 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:31.960 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:31.960 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:31.960 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:31.960 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:31.961 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:31.961 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:31.961 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:31.961 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:31.961 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:31.962 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:31.962 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:31.962 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:31.962 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:31.963 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:31.963 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:31.963 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:31.963 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:31.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:32.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:32.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:32.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:32.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:32.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:32.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:32.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:32.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:32.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:32.430 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:32.430 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:32.430 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:32.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:32.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:32.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:32.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:32.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:32.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:32.938 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:32.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:32.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:33.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368269 2021-05-12T12:57:33.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368269 '!=' null 2021-05-12T12:57:33.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:33.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:33.427 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:33.427 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:33.519 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:33.519 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:33.519 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:33.520 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:33.520 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:33.520 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:33.520 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:33.521 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:33.521 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:33.521 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:33.521 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:33.521 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:33.522 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:33.522 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:33.522 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:33.522 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:33.523 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:33.523 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:33.523 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:33.523 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:33.523 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:33.524 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:33.524 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:33.524 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:33.524 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:33.524 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:33.525 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:33.525 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:33.525 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:33.525 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:33.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:33.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:33.628 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:33.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:33.629 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:33.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:33.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:33.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:33.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:33.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:33.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:33.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:33.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:34.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:34.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:34.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:34.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:34.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:34.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:34.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:34.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:34.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:35.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989202 2021-05-12T12:57:35.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989202 '!=' null 2021-05-12T12:57:35.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:35.065 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:35.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:35.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:35.158 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:35.158 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:35.159 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:35.159 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:35.159 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:35.159 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:35.160 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:35.160 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:35.160 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:35.160 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:35.160 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:35.161 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:35.161 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:35.161 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:35.161 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:35.162 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:35.162 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:35.162 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:35.162 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:35.163 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:35.163 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:35.163 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:35.163 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:35.163 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:35.164 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:35.164 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:35.164 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:35.164 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:35.165 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:35.165 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:35.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:35.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:35.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:35.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:35.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:35.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:35.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:35.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:35.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:35.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:35.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:35.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:35.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:36.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:36.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:36.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:36.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:36.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:36.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:36.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:36.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:36.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:36.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368263 2021-05-12T12:57:36.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368263 '!=' null 2021-05-12T12:57:36.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:36.700 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:36.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:36.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:36.794 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:36.794 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:36.794 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:36.795 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:36.795 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:36.795 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:36.795 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:36.795 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:36.796 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:36.796 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:36.796 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:36.796 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:36.797 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:36.797 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:36.797 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:36.797 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:36.797 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:36.798 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:36.798 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:36.798 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:36.798 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:36.798 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:36.799 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:36.799 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:36.799 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:36.799 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:36.800 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:36.800 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:36.800 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:36.800 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:36.802 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:36.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:36.904 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:36.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:36.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:36.905 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:36.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:36.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:36.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:37.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:37.267 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:37.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:37.268 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:37.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:37.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:37.752 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:37.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:37.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:38.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368265 2021-05-12T12:57:38.239 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368265 '!=' null 2021-05-12T12:57:38.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:38.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:38.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:38.240 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:38.333 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:38.334 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:38.334 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:38.334 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:38.334 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:38.335 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:38.335 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:38.335 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:38.335 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:38.335 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:38.336 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:38.336 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:38.336 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:38.336 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:38.337 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:38.337 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:38.337 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:38.337 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:38.337 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:38.338 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:38.338 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:38.338 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:38.338 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:38.339 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:38.339 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:38.339 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:38.339 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:38.339 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:38.340 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:38.340 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:38.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:38.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:38.444 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:38.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:38.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:38.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:38.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:38.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:38.445 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:38.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:38.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:38.802 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:38.802 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:39.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:39.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:39.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:39.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:39.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:39.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:39.294 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:39.294 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:39.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:39.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989239 2021-05-12T12:57:39.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989239 '!=' null 2021-05-12T12:57:39.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:39.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:39.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:39.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:39.885 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:39.885 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:39.885 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:39.885 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:39.886 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:39.886 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:39.886 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:39.886 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:39.887 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:39.887 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:39.887 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:39.887 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:39.888 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:39.888 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:39.888 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:39.888 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:39.888 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:39.889 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:39.889 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:39.889 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:39.889 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:39.889 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:39.890 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:39.890 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:39.890 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:39.890 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:39.891 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:39.891 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:39.891 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:39.891 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:39.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:39.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:39.995 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:39.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:39.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:39.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:39.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:39.996 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:39.997 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:40.355 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:40.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:40.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:40.356 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:40.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:40.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:40.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:40.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:40.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:40.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:40.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:40.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:40.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:41.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989239 2021-05-12T12:57:41.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989239 '!=' null 2021-05-12T12:57:41.333 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:41.334 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:41.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:41.334 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:41.427 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:41.427 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:41.428 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:41.428 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:41.428 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:41.428 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:41.429 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:41.429 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:41.429 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:41.429 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:41.429 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:41.430 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:41.430 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:41.430 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:41.430 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:41.430 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:41.431 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:41.431 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:41.431 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:41.431 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:41.431 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:41.432 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:41.432 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:41.432 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:41.432 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:41.432 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:41.433 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:41.433 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:41.433 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:41.433 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:41.435 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:41.537 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:41.538 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:41.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:41.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:41.538 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:41.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:41.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:41.539 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:41.898 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:41.898 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:41.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:41.899 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:42.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:42.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:42.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:42.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:42.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:42.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:42.389 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:42.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:42.390 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:42.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241923 2021-05-12T12:57:42.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241923 '!=' null 2021-05-12T12:57:42.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:42.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:42.891 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:42.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:42.984 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:42.984 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:42.985 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:42.985 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:42.985 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:42.985 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:42.986 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:42.986 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:42.986 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:42.986 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:42.986 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:42.987 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:42.987 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:42.987 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:42.987 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:42.988 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:42.988 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:42.988 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:42.988 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:42.989 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:42.989 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:42.989 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:42.989 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:42.989 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:42.990 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:42.990 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:42.990 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:42.990 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:42.990 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:42.991 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:42.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:43.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:43.096 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:43.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:43.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:43.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:43.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:43.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:43.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:43.455 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:43.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:43.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:43.456 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:43.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:43.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:43.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:43.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:43.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:43.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:43.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:43.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:43.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:44.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862858 2021-05-12T12:57:44.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862858 '!=' null 2021-05-12T12:57:44.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:44.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:44.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:44.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:44.573 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:44.574 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:44.574 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:44.574 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:44.574 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:44.575 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:44.575 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:44.575 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:44.575 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:44.575 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:44.576 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:44.576 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:44.576 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:44.576 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:44.576 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:44.577 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:44.577 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:44.577 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:44.577 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:44.578 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:44.578 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:44.578 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:44.578 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:44.578 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:44.579 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:44.579 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:44.579 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:44.579 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:44.580 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:44.580 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:44.581 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:44.683 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:44.684 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:44.684 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:44.684 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:44.685 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:44.685 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:44.685 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:44.685 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:45.039 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:45.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:45.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:45.040 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:45.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:45.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:45.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:45.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:45.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:45.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:45.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:45.522 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:45.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:46.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241923 2021-05-12T12:57:46.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241923 '!=' null 2021-05-12T12:57:46.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:46.014 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:46.014 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:46.014 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:46.109 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:46.110 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:46.110 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:46.110 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:46.110 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:46.111 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:46.111 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:46.111 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:46.111 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:46.112 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:46.112 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:46.112 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:46.112 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:46.112 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:46.113 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:46.113 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:46.113 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:46.113 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:46.114 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:46.114 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:46.114 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:46.114 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:46.114 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:46.115 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:46.115 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:46.115 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:46.115 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:46.116 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:46.116 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:46.116 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:46.117 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:46.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:46.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:46.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:46.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:46.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:46.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:46.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:46.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:46.576 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:46.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:46.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:46.577 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:47.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:47.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:47.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:47.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:47.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:47.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:47.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:47.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:47.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:47.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241919 2021-05-12T12:57:47.594 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241919 '!=' null 2021-05-12T12:57:47.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:47.595 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:47.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:47.595 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:47.688 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:47.689 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:47.689 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:47.689 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:47.690 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:47.690 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:47.690 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:47.690 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:47.690 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:47.691 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:47.691 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:47.691 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:47.691 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:47.692 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:47.692 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:47.692 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:47.692 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:47.693 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:47.693 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:47.693 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:47.693 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:47.694 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:47.694 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:47.694 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:47.694 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:47.695 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:47.695 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:47.695 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:47.695 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:47.695 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:47.697 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:47.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:47.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:47.799 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:47.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:47.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:47.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:47.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:47.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:48.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:48.160 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:48.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:48.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:48.643 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:48.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:48.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:48.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:48.644 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:48.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:48.645 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:48.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:48.645 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:49.132 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241919 2021-05-12T12:57:49.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241919 '!=' null 2021-05-12T12:57:49.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:49.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:49.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:49.134 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:49.226 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:49.226 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:49.226 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:49.226 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:49.227 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:49.227 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:49.227 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:49.227 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:49.227 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:49.228 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:49.228 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:49.228 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:49.228 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:49.228 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:49.229 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:49.229 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:49.229 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:49.229 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:49.229 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:49.230 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:49.230 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:49.230 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:49.230 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:49.231 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:49.231 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:49.231 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:49.231 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:49.231 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:49.232 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:49.232 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:49.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:49.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:49.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:49.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:49.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:49.336 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:49.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:49.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:49.337 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:49.698 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:49.699 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:49.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:49.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:50.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:50.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:50.186 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:50.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:50.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:50.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:50.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:50.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:50.188 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:50.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862822 2021-05-12T12:57:50.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862822 '!=' null 2021-05-12T12:57:50.676 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:50.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:50.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:50.677 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:50.771 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:50.771 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:50.772 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:50.772 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:50.772 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:50.772 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:50.773 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:50.773 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:50.773 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:50.773 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:50.774 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:50.774 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:50.774 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:50.774 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:50.774 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:50.775 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:50.775 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:50.775 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:50.775 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:50.776 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:50.776 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:50.776 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:50.776 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:50.776 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:50.777 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:50.777 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:50.777 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:50.777 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:50.778 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:50.778 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:50.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:50.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:50.881 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:50.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:50.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:50.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:50.882 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:50.882 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:50.882 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:51.252 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:51.252 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:51.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:51.253 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:51.739 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:51.739 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:51.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:51.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:51.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:51.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:51.740 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:51.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:51.741 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:52.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241820 2021-05-12T12:57:52.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241820 '!=' null 2021-05-12T12:57:52.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:52.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:52.220 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:52.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:52.312 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:52.312 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:52.313 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:52.313 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:52.313 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:52.313 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:52.314 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:52.314 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:52.314 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:52.314 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:52.315 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:52.315 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:52.315 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:52.315 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:52.315 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:52.316 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:52.316 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:52.316 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:52.316 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:52.316 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:52.317 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:52.317 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:52.317 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:52.317 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:52.317 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:52.318 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:52.318 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:52.318 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:52.318 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:52.319 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:52.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:52.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:52.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:52.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:52.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:52.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:52.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:52.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:52.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:52.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:52.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:52.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:52.782 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:53.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:53.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:53.267 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:53.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:53.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:53.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:53.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:53.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:53.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:53.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862762 2021-05-12T12:57:53.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862762 '!=' null 2021-05-12T12:57:53.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:53.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:53.765 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:53.765 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:53.858 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:53.858 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:53.858 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:53.859 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:53.859 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:53.859 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:53.859 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:53.860 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:53.860 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:53.860 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:53.860 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:53.860 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:53.861 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:53.861 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:53.861 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:53.861 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:53.861 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:53.862 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:53.862 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:53.862 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:53.862 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:53.863 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:53.863 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:53.863 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:53.863 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:53.864 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:53.864 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:53.864 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:53.864 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:53.864 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:53.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:53.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:53.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:53.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:53.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:53.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:53.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:53.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:53.969 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:54.330 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:54.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:54.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:54.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:54.826 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:54.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:54.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:54.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:54.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:54.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:54.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:54.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:54.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:55.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862762 2021-05-12T12:57:55.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862762 '!=' null 2021-05-12T12:57:55.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:55.308 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:55.308 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:55.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:55.402 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:55.403 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:55.403 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:55.403 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:55.403 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:55.403 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:55.404 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:55.404 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:55.404 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:55.404 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:55.404 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:55.405 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:55.405 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:55.405 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:55.405 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:55.406 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:55.406 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:55.406 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:55.406 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:55.406 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:55.407 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:55.407 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:55.407 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:55.407 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:55.407 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:55.408 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:55.408 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:55.408 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:55.408 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:55.409 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:55.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:55.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:55.512 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:55.513 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:55.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:55.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:55.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:55.514 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:55.515 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:55.875 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:55.875 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:55.875 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:55.876 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:56.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:56.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:56.381 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:56.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:56.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:56.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:56.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:56.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:56.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:56.866 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241834 2021-05-12T12:57:56.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241834 '!=' null 2021-05-12T12:57:56.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:56.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:56.867 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:56.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:56.961 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:56.962 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:56.962 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:56.962 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:56.962 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:56.963 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:56.964 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:56.965 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:56.966 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:56.966 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:56.966 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:56.966 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:56.967 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:56.968 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:56.970 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:57.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:57.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:57.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:57.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:57.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:57.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:57.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:57.073 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:57.433 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:57.433 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:57.434 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:57.434 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:57.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:57.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:57.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:57.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:57.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:57.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:57.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:57.991 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:57.991 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:57:58.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241796 2021-05-12T12:57:58.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241796 '!=' null 2021-05-12T12:57:58.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:57:58.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:58.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:57:58.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:58.592 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:57:58.592 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:57:58.592 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:58.592 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:58.593 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:58.593 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:57:58.593 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:58.593 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:57:58.593 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:58.594 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:58.594 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:58.594 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:58.594 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:57:58.595 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:57:58.595 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:58.595 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:58.595 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:57:58.595 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:57:58.596 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:57:58.596 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:57:58.596 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:58.596 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:57:58.596 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:57:58.597 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:57:58.597 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:57:58.597 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:57:58.597 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:57:58.597 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:57:58.598 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:57:58.598 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:57:58.600 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:57:58.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:57:58.701 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:57:58.702 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:57:58.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:57:58.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:57:58.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:57:58.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:57:58.703 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:57:59.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:57:59.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:57:59.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:57:59.061 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:57:59.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:57:59.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:57:59.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:57:59.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:57:59.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:57:59.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:57:59.552 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:57:59.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:57:59.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:00.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862657 2021-05-12T12:58:00.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862657 '!=' null 2021-05-12T12:58:00.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:00.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:00.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:00.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:00.188 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:00.188 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:00.188 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:00.188 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:00.189 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:00.189 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:00.189 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:00.189 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:00.190 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:00.190 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:00.190 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:00.190 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:00.190 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:00.191 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:00.191 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:00.191 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:00.191 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:00.192 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:00.192 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:00.192 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:00.192 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:00.193 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:00.193 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:00.193 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:00.193 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:00.193 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:00.194 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:00.194 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:00.194 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:00.194 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:00.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:00.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:00.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:00.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:00.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:00.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:00.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:00.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:00.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:00.656 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:00.656 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:00.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:00.657 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:01.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:01.164 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:01.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:01.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:01.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:01.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:01.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:01.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:01.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:01.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241823 2021-05-12T12:58:01.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241823 '!=' null 2021-05-12T12:58:01.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:01.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:01.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:01.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:01.763 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:01.764 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:01.764 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:01.764 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:01.764 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:01.765 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:01.765 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:01.765 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:01.765 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:01.765 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:01.766 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:01.766 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:01.766 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:01.766 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:01.767 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:01.767 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:01.767 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:01.767 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:01.768 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:01.768 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:01.768 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:01.768 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:01.768 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:01.769 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:01.769 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:01.769 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:01.769 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:01.769 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:01.770 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:01.770 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:01.771 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:01.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:01.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:01.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:01.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:01.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:01.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:01.875 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:01.875 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:02.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:02.241 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:02.241 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:02.241 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:02.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:02.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:02.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:02.730 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:02.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:02.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:02.731 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:02.731 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:02.732 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:03.217 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241823 2021-05-12T12:58:03.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241823 '!=' null 2021-05-12T12:58:03.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:03.218 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:03.218 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:03.219 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:03.312 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:03.312 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:03.312 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:03.312 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:03.313 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:03.313 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:03.313 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:03.313 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:03.314 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:03.314 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:03.314 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:03.314 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:03.314 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:03.315 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:03.315 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:03.315 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:03.315 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:03.316 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:03.316 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:03.316 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:03.316 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:03.316 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:03.317 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:03.317 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:03.317 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:03.317 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:03.318 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:03.318 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:03.318 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:03.318 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:03.320 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:03.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:03.422 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:03.422 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:03.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:03.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:03.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:03.423 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:03.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:03.785 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:03.785 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:03.786 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:03.786 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:04.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:04.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:04.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:04.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:04.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:04.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:04.270 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:04.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:04.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:04.756 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862824 2021-05-12T12:58:04.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862824 '!=' null 2021-05-12T12:58:04.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:04.757 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:04.758 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:04.758 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:04.853 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:04.853 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:04.854 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:04.854 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:04.854 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:04.854 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:04.855 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:04.855 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:04.855 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:04.855 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:04.856 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:04.856 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:04.856 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:04.856 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:04.857 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:04.857 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:04.857 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:04.857 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:04.857 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:04.858 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:04.858 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:04.858 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:04.858 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:04.859 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:04.859 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:04.859 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:04.859 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:04.859 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:04.860 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:04.860 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:04.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:04.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:04.963 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:04.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:04.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:04.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:04.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:04.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:04.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:05.331 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:05.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:05.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:05.331 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:05.817 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:05.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:05.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:05.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:05.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:05.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:05.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:05.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:05.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:06.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241827 2021-05-12T12:58:06.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241827 '!=' null 2021-05-12T12:58:06.324 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:06.325 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:06.325 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:06.325 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:06.418 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:06.419 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:06.419 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:06.419 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:06.419 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:06.420 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:06.420 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:06.420 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:06.420 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:06.420 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:06.421 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:06.421 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:06.421 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:06.421 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:06.422 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:06.422 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:06.422 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:06.422 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:06.422 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:06.423 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:06.423 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:06.423 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:06.423 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:06.423 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:06.424 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:06.424 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:06.424 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:06.424 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:06.425 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:06.425 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:06.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:06.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:06.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:06.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:06.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:06.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:06.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:06.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:06.530 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:06.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:06.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:06.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:06.897 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:07.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:07.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:07.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:07.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:07.385 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:07.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:07.386 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:07.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:07.386 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:07.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241814 2021-05-12T12:58:07.879 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241814 '!=' null 2021-05-12T12:58:07.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:07.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:07.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:07.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:07.973 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:07.973 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:07.974 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:07.974 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:07.974 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:07.974 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:07.975 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:07.975 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:07.975 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:07.975 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:07.975 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:07.976 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:07.976 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:07.976 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:07.976 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:07.977 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:07.977 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:07.977 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:07.977 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:07.977 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:07.978 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:07.978 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:07.978 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:07.978 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:07.979 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:07.979 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:07.979 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:07.979 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:07.979 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:07.980 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:07.981 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:08.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:08.083 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:08.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:08.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:08.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:08.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:08.084 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:08.085 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:08.443 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:08.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:08.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:08.444 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:08.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:08.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:08.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:08.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:08.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:08.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:08.937 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:08.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:08.938 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:09.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862723 2021-05-12T12:58:09.426 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862723 '!=' null 2021-05-12T12:58:09.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:09.427 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:09.427 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:09.428 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:09.520 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:09.520 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:09.521 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:09.521 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:09.521 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:09.521 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:09.521 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:09.522 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:09.522 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:09.522 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:09.522 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:09.523 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:09.523 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:09.523 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:09.523 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:09.525 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:09.525 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:09.525 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:09.525 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:09.526 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:09.526 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:09.526 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:09.526 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:09.527 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:09.527 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:09.528 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:09.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:09.630 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:09.630 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:09.631 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:09.631 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:09.631 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:09.631 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:09.631 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:09.989 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:09.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:09.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:09.990 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:10.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:10.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:10.478 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:10.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:10.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:10.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:10.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:10.479 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:10.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:11.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862723 2021-05-12T12:58:11.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862723 '!=' null 2021-05-12T12:58:11.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:11.139 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:11.139 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:11.139 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:11.232 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:11.232 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:11.232 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:11.233 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:11.233 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:11.233 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:11.233 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:11.233 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:11.234 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:11.234 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:11.234 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:11.234 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:11.235 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:11.235 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:11.235 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:11.235 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:11.235 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:11.236 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:11.236 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:11.236 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:11.237 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:11.237 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:11.237 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:11.237 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:11.237 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:11.238 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:11.238 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:11.238 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:11.238 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:11.238 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:11.240 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:11.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:11.342 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:11.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:11.343 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:11.344 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:11.344 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:11.344 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:11.344 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:11.823 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:11.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:11.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:11.824 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:12.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:12.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:12.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:12.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:12.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:12.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:12.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:12.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:12.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:12.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241769 2021-05-12T12:58:12.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241769 '!=' null 2021-05-12T12:58:12.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:12.805 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:12.805 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:12.805 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:12.898 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:12.898 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:12.899 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:12.899 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:12.899 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:12.899 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:12.899 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:12.900 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:12.900 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:12.900 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:12.900 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:12.901 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:12.901 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:12.901 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:12.901 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:12.902 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:12.902 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:12.902 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:12.902 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:12.903 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:12.903 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:12.903 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:12.903 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:12.904 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:12.904 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:12.904 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:12.904 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:12.904 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:12.905 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:12.905 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:12.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:13.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:13.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:13.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:13.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:13.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:13.009 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:13.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:13.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:13.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:13.371 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:13.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:13.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:13.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:13.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:13.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:13.873 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:13.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:13.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:13.874 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:13.875 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:13.875 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:14.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989023 2021-05-12T12:58:14.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989023 '!=' null 2021-05-12T12:58:14.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:14.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:14.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:14.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:14.452 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:14.452 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:14.453 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:14.453 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:14.453 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:14.453 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:14.453 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:14.454 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:14.454 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:14.454 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:14.454 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:14.455 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:14.455 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:14.455 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:14.455 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:14.455 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:14.456 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:14.456 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:14.456 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:14.456 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:14.456 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:14.457 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:14.457 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:14.457 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:14.457 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:14.458 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:14.458 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:14.458 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:14.458 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:14.458 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:14.461 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:14.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:14.563 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:14.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:14.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:14.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:14.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:14.564 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:14.565 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:14.919 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:14.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:14.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:14.920 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:15.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:15.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:15.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:15.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:15.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:15.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:15.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:15.409 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:15.409 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:15.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368127 2021-05-12T12:58:15.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368127 '!=' null 2021-05-12T12:58:15.910 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:15.911 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:15.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:15.911 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:16.003 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:16.004 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:16.004 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:16.004 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:16.004 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:16.005 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:16.005 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:16.005 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:16.005 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:16.005 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:16.006 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:16.006 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:16.006 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:16.006 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:16.006 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:16.007 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:16.007 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:16.007 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:16.007 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:16.008 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:16.008 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:16.008 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:16.008 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:16.008 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:16.009 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:16.009 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:16.009 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:16.009 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:16.010 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:16.010 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:16.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:16.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:16.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:16.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:16.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:16.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:16.114 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:16.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:16.115 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:16.480 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:16.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:16.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:16.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:17.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:17.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:17.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:17.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:17.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:17.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:17.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:17.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:17.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:17.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368129 2021-05-12T12:58:17.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368129 '!=' null 2021-05-12T12:58:17.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:17.550 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:17.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:17.550 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:17.644 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:17.644 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:17.644 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:17.644 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:17.645 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:17.645 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:17.645 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:17.645 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:17.645 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:17.646 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:17.646 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:17.646 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:17.646 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:17.647 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:17.647 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:17.647 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:17.647 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:17.648 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:17.648 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:17.648 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:17.648 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:17.649 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:17.649 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:17.649 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:17.649 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:17.649 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:17.650 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:17.650 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:17.650 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:17.650 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:17.652 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:17.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:17.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:17.754 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:17.754 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:17.754 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:17.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:17.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:17.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:18.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:18.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:18.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:18.111 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:18.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:18.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:18.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:18.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:18.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:18.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:18.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:18.607 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:18.607 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:19.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368129 2021-05-12T12:58:19.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368129 '!=' null 2021-05-12T12:58:19.182 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:19.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:19.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:19.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:19.277 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:19.277 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:19.278 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:19.278 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:19.279 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:19.279 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:19.280 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:19.280 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:19.280 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:19.281 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:19.281 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:19.281 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:19.281 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:19.282 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:19.282 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:19.282 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:19.283 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:19.284 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:19.284 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:19.284 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:19.284 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:19.285 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:19.285 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:19.285 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:19.285 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:19.285 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:19.286 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:19.387 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:19.388 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:19.388 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:19.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:19.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:19.389 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:19.390 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:19.390 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:19.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:19.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:19.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:19.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:20.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:20.231 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:20.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:20.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:20.232 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:20.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:20.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:20.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:20.233 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:20.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989049 2021-05-12T12:58:20.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989049 '!=' null 2021-05-12T12:58:20.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:20.727 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:20.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:20.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:20.820 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:20.820 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:20.820 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:20.821 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:20.821 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:20.821 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:20.821 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:20.822 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:20.822 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:20.822 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:20.822 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:20.823 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:20.823 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:20.823 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:20.823 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:20.823 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:20.824 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:20.824 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:20.824 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:20.824 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:20.824 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:20.825 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:20.825 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:20.825 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:20.825 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:20.826 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:20.826 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:20.826 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:20.826 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:20.826 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:20.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:20.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:20.930 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:20.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:20.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:20.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:20.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:20.931 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:20.932 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:21.291 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:21.291 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:21.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:21.292 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:21.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:21.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:21.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:21.779 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:21.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:21.780 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:21.781 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:21.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:21.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:22.262 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368187 2021-05-12T12:58:22.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368187 '!=' null 2021-05-12T12:58:22.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:22.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:22.263 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:22.263 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:22.357 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:22.357 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:22.357 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:22.358 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:22.358 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:22.358 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:22.358 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:22.358 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:22.359 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:22.359 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:22.359 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:22.359 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:22.360 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:22.360 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:22.360 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:22.360 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:22.360 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:22.361 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:22.361 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:22.361 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:22.361 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:22.362 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:22.362 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:22.362 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:22.362 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:22.362 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:22.363 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:22.363 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:22.363 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:22.363 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:22.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:22.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:22.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:22.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:22.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:22.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:22.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:22.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:22.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:22.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:22.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:22.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:22.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:23.317 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:23.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:23.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:23.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:23.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:23.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:23.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:23.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:23.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:23.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989198 2021-05-12T12:58:23.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989198 '!=' null 2021-05-12T12:58:23.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:23.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:23.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:23.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:23.923 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:23.923 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:23.924 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:23.924 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:23.924 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:23.924 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:23.924 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:23.925 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:23.925 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:23.925 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:23.925 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:23.926 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:23.926 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:23.926 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:23.926 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:23.927 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:23.927 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:23.927 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:23.927 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:23.927 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:23.928 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:23.928 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:23.928 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:23.928 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:23.929 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:23.929 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:23.929 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:23.929 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:23.929 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:23.930 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:23.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:24.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:24.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:24.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:24.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:24.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:24.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:24.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:24.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:24.392 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:24.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:24.393 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:24.394 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:24.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:24.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:24.882 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:24.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:24.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:24.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:24.883 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:24.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:24.884 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:25.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989198 2021-05-12T12:58:25.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989198 '!=' null 2021-05-12T12:58:25.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:25.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:25.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:25.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:25.493 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:25.493 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:25.493 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:25.494 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:25.494 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:25.494 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:25.494 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:25.495 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:25.495 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:25.495 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:25.495 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:25.496 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:25.496 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:25.496 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:25.496 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:25.497 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:25.497 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:25.497 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:25.497 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:25.497 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:25.498 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:25.498 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:25.498 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:25.498 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:25.499 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:25.499 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:25.499 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:25.499 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:25.500 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:25.500 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:25.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:25.603 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:25.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:25.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:25.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:25.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:25.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:25.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:25.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:25.964 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:25.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:25.964 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:25.965 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:26.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:26.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:26.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:26.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:26.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:26.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:26.470 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:26.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:26.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:26.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241888 2021-05-12T12:58:26.977 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241888 '!=' null 2021-05-12T12:58:26.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:26.978 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:26.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:26.978 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:27.071 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:27.072 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:27.072 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:27.072 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:27.072 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:27.073 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:27.073 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:27.073 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:27.073 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:27.074 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:27.074 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:27.074 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:27.074 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:27.075 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:27.075 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:27.075 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:27.075 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:27.075 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:27.076 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:27.076 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:27.076 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:27.076 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:27.077 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:27.077 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:27.077 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:27.077 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:27.077 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:27.078 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:27.078 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:27.078 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:27.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:27.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:27.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:27.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:27.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:27.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:27.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:27.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:27.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:27.542 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:27.542 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:27.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:27.543 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:28.031 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:28.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:28.032 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:28.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:28.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:28.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:28.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:28.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:28.035 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:28.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241912 2021-05-12T12:58:28.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241912 '!=' null 2021-05-12T12:58:28.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:28.526 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:28.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:28.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:28.619 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:28.620 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:28.620 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:28.620 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:28.621 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:28.621 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:28.622 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:28.622 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:28.622 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:28.623 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:28.623 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:28.624 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:28.624 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:28.625 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:28.625 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:28.626 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:28.626 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:28.627 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:28.627 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:28.628 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:28.628 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:28.628 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:28.629 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:28.629 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:28.630 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:28.630 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:28.631 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:28.631 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:28.632 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:28.632 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:28.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:28.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:28.729 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:28.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:28.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:28.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:28.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:28.730 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:28.731 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:29.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:29.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:29.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:29.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:29.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:29.585 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:29.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:29.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:29.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:29.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:29.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:29.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:29.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:30.072 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862918 2021-05-12T12:58:30.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862918 '!=' null 2021-05-12T12:58:30.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:30.073 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:30.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:30.074 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:30.167 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:30.167 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:30.168 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:30.168 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:30.168 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:30.168 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:30.169 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:30.169 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:30.169 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:30.169 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:30.169 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:30.170 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:30.170 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:30.170 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:30.170 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:30.171 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:30.171 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:30.171 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:30.172 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:30.172 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:30.172 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:30.172 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:30.172 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:30.173 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:30.173 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:30.173 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:30.173 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:30.174 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:30.174 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:30.174 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:30.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:30.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:30.277 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:30.277 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:30.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:30.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:30.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:30.278 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:30.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:30.637 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:30.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:30.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:30.638 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:31.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:31.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:31.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:31.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:31.124 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:31.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:31.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:31.125 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:31.125 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:31.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241936 2021-05-12T12:58:31.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241936 '!=' null 2021-05-12T12:58:31.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:31.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:31.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:31.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:31.716 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:31.716 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:31.717 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:31.717 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:31.717 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:31.717 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:31.717 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:31.718 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:31.718 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:31.718 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:31.718 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:31.718 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:31.719 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:31.719 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:31.719 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:31.719 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:31.719 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:31.720 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:31.720 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:31.720 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:31.720 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:31.720 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:31.721 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:31.721 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:31.721 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:31.721 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:31.721 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:31.722 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:31.722 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:31.722 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:31.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:31.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:31.827 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:31.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:31.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:31.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:31.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:31.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:31.830 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:32.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:32.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:32.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:32.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:32.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:32.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:32.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:32.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:32.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:32.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:32.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:32.667 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:32.668 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:33.180 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241936 2021-05-12T12:58:33.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241936 '!=' null 2021-05-12T12:58:33.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:33.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:33.181 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:33.182 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:33.274 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:33.274 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:33.274 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:33.274 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:33.275 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:33.275 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:33.275 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:33.275 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:33.276 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:33.276 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:33.276 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:33.276 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:33.276 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:33.277 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:33.277 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:33.277 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:33.277 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:33.277 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:33.278 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:33.278 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:33.278 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:33.278 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:33.279 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:33.279 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:33.279 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:33.279 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:33.279 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:33.280 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:33.280 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:33.280 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:33.282 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:33.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:33.384 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:33.384 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:33.384 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:33.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:33.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:33.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:33.385 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:33.733 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:33.734 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:33.734 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:33.734 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:34.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:34.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:34.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:34.210 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:34.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:34.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:34.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:34.211 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:34.212 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:34.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862901 2021-05-12T12:58:34.808 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862901 '!=' null 2021-05-12T12:58:34.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:34.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:34.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:34.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:34.901 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:34.901 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:34.901 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:34.901 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:34.902 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:34.902 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:34.902 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:34.902 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:34.902 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:34.903 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:34.903 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:34.903 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:34.903 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:34.903 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:34.904 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:34.904 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:34.904 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:34.904 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:34.904 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:34.904 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:34.905 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:34.905 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:34.905 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:34.905 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:34.905 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:34.906 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:34.906 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:34.906 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:34.906 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:34.907 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:34.909 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:35.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:35.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:35.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:35.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:35.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:35.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:35.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:35.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:35.365 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:35.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:35.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:35.366 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:35.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:35.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:35.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:35.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:35.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:35.849 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:35.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:35.850 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:35.850 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:36.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241918 2021-05-12T12:58:36.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241918 '!=' null 2021-05-12T12:58:36.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:36.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:36.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:36.377 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:36.468 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:36.468 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:36.469 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:36.469 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:36.469 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:36.469 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:36.470 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:36.470 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:36.470 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:36.471 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:36.471 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:36.471 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:36.471 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:36.472 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:36.472 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:36.472 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:36.472 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:36.473 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:36.473 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:36.473 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:36.473 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:36.474 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:36.474 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:36.474 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:36.474 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:36.474 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:36.475 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:36.475 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:36.475 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:36.475 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:36.476 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:36.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:36.578 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:36.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:36.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:36.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:36.579 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:36.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:36.580 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:36.936 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:36.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:36.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:36.937 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:37.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:37.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:37.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:37.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:37.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:37.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:37.414 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:37.415 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:37.415 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:37.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241951 2021-05-12T12:58:37.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241951 '!=' null 2021-05-12T12:58:37.894 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:37.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:37.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:37.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:37.998 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:37.999 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:37.999 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:37.999 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:37.999 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:38.000 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:38.000 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:38.000 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:38.000 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:38.001 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:38.001 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:38.001 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:38.001 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:38.001 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:38.002 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:38.002 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:38.002 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:38.002 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:38.003 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:38.003 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:38.003 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:38.003 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:38.003 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:38.004 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:38.004 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:38.004 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:38.004 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:38.005 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:38.005 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:38.005 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:38.006 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:38.108 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:38.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:38.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:38.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:38.109 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:38.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:38.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:38.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:38.460 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:38.461 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:38.461 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:38.461 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:38.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:38.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:38.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:38.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:38.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:38.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:38.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:38.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:38.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:39.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241951 2021-05-12T12:58:39.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241951 '!=' null 2021-05-12T12:58:39.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:39.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:39.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:39.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:39.548 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:39.548 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:39.549 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:39.549 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:39.549 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:39.549 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:39.550 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:39.550 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:39.550 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:39.550 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:39.550 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:39.551 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:39.551 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:39.551 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:39.551 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:39.552 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:39.552 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:39.552 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:39.552 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:39.552 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:39.553 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:39.553 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:39.553 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:39.553 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:39.553 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:39.554 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:39.554 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:39.554 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:39.554 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:39.555 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:39.557 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:39.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:39.658 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:39.659 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:39.660 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:39.660 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:39.660 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:39.660 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:39.661 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:40.019 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:40.019 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:40.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:40.020 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:40.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:40.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:40.504 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:40.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:40.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:40.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:40.505 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:40.506 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:40.506 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:40.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862913 2021-05-12T12:58:40.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862913 '!=' null 2021-05-12T12:58:40.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:40.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:40.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:40.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:41.080 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:41.080 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:41.081 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:41.081 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:41.082 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:41.082 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:41.082 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:41.083 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:41.083 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:41.084 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:41.084 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:41.085 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:41.085 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:41.085 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:41.086 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:41.086 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:41.087 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:41.087 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:41.088 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:41.088 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:41.088 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:41.089 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:41.089 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:41.090 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:41.090 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:41.091 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:41.091 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:41.091 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:41.092 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:41.092 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:41.093 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:41.190 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:41.191 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:41.191 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:41.192 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:41.192 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:41.192 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:41.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:41.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:41.592 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:41.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:41.592 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:41.593 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:42.077 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:42.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:42.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:42.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:42.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:42.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:42.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:42.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:42.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:42.561 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241943 2021-05-12T12:58:42.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241943 '!=' null 2021-05-12T12:58:42.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:42.562 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:42.562 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:42.563 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:42.655 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:42.656 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:42.656 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:42.656 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:42.656 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:42.657 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:42.657 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:42.657 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:42.657 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:42.658 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:42.658 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:42.658 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:42.658 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:42.658 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:42.659 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:42.659 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:42.659 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:42.659 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:42.660 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:42.660 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:42.660 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:42.660 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:42.660 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:42.661 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:42.661 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:42.661 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:42.661 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:42.662 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:42.662 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:42.662 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:42.664 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:42.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:42.766 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:42.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:42.766 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:42.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:42.767 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:42.768 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:42.768 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:43.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:43.120 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:43.121 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:43.121 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:43.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:43.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:43.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:43.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:43.625 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:43.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:43.626 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:43.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:43.627 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:44.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862945 2021-05-12T12:58:44.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862945 '!=' null 2021-05-12T12:58:44.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:44.111 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:44.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:44.112 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:44.204 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:44.204 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:44.204 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:44.205 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:44.205 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:44.205 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:44.205 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:44.206 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:44.206 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:44.206 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:44.206 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:44.206 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:44.207 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:44.207 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:44.207 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:44.207 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:44.208 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:44.208 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:44.208 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:44.208 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:44.209 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:44.209 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:44.209 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:44.209 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:44.210 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:44.210 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:44.210 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:44.210 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:44.211 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:44.211 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:44.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:44.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:44.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:44.314 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:44.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:44.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:44.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:44.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:44.316 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:44.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:44.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:44.672 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:44.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:45.187 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:45.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:45.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:45.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:45.188 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:45.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:45.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:45.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:45.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:45.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241959 2021-05-12T12:58:45.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241959 '!=' null 2021-05-12T12:58:45.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:45.675 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:45.675 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:45.675 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:45.767 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:45.768 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:45.768 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:45.768 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:45.769 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:45.769 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:45.769 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:45.769 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:45.769 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:45.770 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:45.770 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:45.770 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:45.770 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:45.770 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:45.771 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:45.771 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:45.771 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:45.771 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:45.772 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:45.772 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:45.772 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:45.772 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:45.772 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:45.773 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:45.773 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:45.773 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:45.773 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:45.774 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:45.774 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:45.774 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:45.776 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:45.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:45.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:45.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:45.878 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:45.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:45.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:45.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:45.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:46.244 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:46.244 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:46.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:46.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:46.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:46.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:46.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:46.790 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:46.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:46.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:46.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:46.791 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:46.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:47.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241959 2021-05-12T12:58:47.274 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241959 '!=' null 2021-05-12T12:58:47.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:47.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:47.275 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:47.275 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:47.368 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:47.369 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:47.369 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:47.369 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:47.370 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:47.370 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:47.370 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:47.370 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:47.371 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:47.371 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:47.371 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:47.371 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:47.371 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:47.372 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:47.372 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:47.372 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:47.373 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:47.373 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:47.373 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:47.374 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:47.374 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:47.374 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:47.374 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:47.375 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:47.375 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:47.375 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:47.375 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:47.375 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:47.376 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:47.376 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:47.377 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:47.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:47.479 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:47.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:47.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:47.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:47.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:47.480 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:47.481 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:47.835 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:47.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:47.836 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:47.837 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:48.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:48.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:48.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:48.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:48.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:48.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:48.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:48.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:48.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:48.852 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241975 2021-05-12T12:58:48.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241975 '!=' null 2021-05-12T12:58:48.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:48.853 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:48.853 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:48.854 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:48.947 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:48.947 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:48.947 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:48.948 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:48.948 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:48.948 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:48.948 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:48.948 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:48.949 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:48.949 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:48.949 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:48.949 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:48.949 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:48.950 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:48.950 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:48.950 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:48.950 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:48.951 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:48.951 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:48.951 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:48.951 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:48.951 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:48.952 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:48.952 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:48.952 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:48.952 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:48.952 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:48.953 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:48.953 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:48.953 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:48.955 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:49.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:49.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:49.057 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:49.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:49.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:49.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:49.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:49.058 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:49.413 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:49.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:49.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:49.414 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:49.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:49.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:49.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:49.895 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:49.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:49.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:49.896 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:49.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:49.896 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:50.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862921 2021-05-12T12:58:50.448 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862921 '!=' null 2021-05-12T12:58:50.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:50.449 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:50.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:50.449 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:50.543 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:50.543 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:50.543 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:50.543 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:50.544 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:50.545 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:50.545 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:50.545 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:50.545 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:50.546 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:50.546 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:50.546 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:50.546 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:50.546 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:50.547 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:50.547 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:50.547 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:50.547 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:50.548 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:50.548 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:50.548 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:50.548 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:50.548 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:50.549 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:50.549 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:50.549 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:50.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:50.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:50.653 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:50.653 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:50.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:50.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:50.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:50.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:50.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:51.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:51.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:51.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:51.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:51.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:51.496 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:51.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:51.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:51.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:51.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:51.498 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:51.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:51.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:51.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241928 2021-05-12T12:58:51.990 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241928 '!=' null 2021-05-12T12:58:51.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:51.991 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:51.991 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:51.991 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:52.082 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:52.083 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:52.083 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:52.083 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:52.084 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:52.084 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:52.085 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:52.085 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:52.085 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:52.086 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:52.086 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:52.086 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:52.087 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:52.087 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:52.087 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:52.087 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:52.088 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:52.088 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:52.088 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:52.088 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:52.089 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:52.089 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:52.089 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:52.089 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:52.089 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:52.090 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:52.090 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:52.090 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:52.091 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:52.091 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:52.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:52.192 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:52.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:52.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:52.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:52.193 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:52.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:52.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:52.194 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:52.551 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:52.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:52.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:52.552 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:53.034 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:53.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:53.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:53.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:53.035 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:53.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:53.036 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:53.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:53.036 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:53.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862913 2021-05-12T12:58:53.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862913 '!=' null 2021-05-12T12:58:53.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:53.572 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:53.572 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:53.573 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:53.669 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:53.669 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:53.669 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:53.669 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:53.670 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:53.670 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:53.670 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:53.670 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:53.671 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:53.671 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:53.671 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:53.671 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:53.672 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:53.672 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:53.672 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:53.672 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:53.673 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:53.673 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:53.673 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:53.673 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:53.673 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:53.674 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:53.674 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:53.674 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:53.674 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:53.675 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:53.675 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:53.675 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:53.675 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:53.676 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:53.677 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:53.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:53.778 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:53.779 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:53.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:53.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:53.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:53.780 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:53.781 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:54.138 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:54.139 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:54.139 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:54.140 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:54.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:54.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:54.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:54.621 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:54.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:54.622 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:54.623 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:54.623 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:54.624 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:55.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862913 2021-05-12T12:58:55.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862913 '!=' null 2021-05-12T12:58:55.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:55.110 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:55.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:55.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:55.204 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:55.205 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:55.205 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:55.205 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:55.205 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:55.206 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:55.206 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:55.206 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:55.206 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:55.206 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:55.207 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:55.207 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:55.207 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:55.207 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:55.208 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:55.208 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:55.208 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:55.208 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:55.209 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:55.209 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:55.209 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:55.209 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:55.209 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:55.210 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:55.210 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:55.210 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:55.210 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:55.211 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:55.211 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:55.211 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:55.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:55.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:55.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:55.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:55.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:55.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:55.316 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:55.316 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:55.316 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:55.705 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:55.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:55.706 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:55.707 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:56.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:56.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:56.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:56.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:56.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:56.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:56.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:56.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:56.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:56.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241852 2021-05-12T12:58:56.681 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241852 '!=' null 2021-05-12T12:58:56.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:56.682 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:56.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:56.682 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:56.779 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:56.779 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:56.779 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:56.779 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:56.780 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:56.780 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:56.780 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:56.780 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:56.781 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:56.781 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:56.781 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:56.781 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:56.782 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:56.782 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:56.782 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:56.782 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:56.782 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:56.783 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:56.783 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:56.783 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:56.783 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:56.784 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:56.784 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:56.784 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:56.784 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:56.784 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:56.785 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:56.785 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:56.785 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:56.785 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:56.787 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:56.889 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:56.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:56.890 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:56.890 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:56.890 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:56.891 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:56.891 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:56.891 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:57.245 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:57.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:57.245 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:57.246 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:57.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:57.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:57.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:57.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:57.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:57.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:57.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:57.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:57.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:58.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241807 2021-05-12T12:58:58.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241807 '!=' null 2021-05-12T12:58:58.234 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:58.235 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:58.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:58.235 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:58.327 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:58.328 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:58.328 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:58.328 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:58.328 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:58.329 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:58.329 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:58.329 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:58.330 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:58.330 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:58.330 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:58.330 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:58.331 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:58.331 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:58.331 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:58.331 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:58.332 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:58.332 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:58.332 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:58.332 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:58.332 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:58.333 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:58.333 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:58.333 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:58.333 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:58.334 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:58.334 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:58.334 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:58.334 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:58.334 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:58.336 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:58.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:58.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:58.438 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:58.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:58.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:58.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:58.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:58.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:58.811 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:58.812 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:58.812 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:58.812 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:59.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:59.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:59.296 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:59.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:59.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:59.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:59.297 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:59.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:59.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:59.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989060 2021-05-12T12:58:59.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989060 '!=' null 2021-05-12T12:58:59.788 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:59.789 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:59.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:59.789 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:59.882 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:58:59.882 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:58:59.882 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:59.883 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:59.883 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:59.883 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:58:59.883 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:59.883 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:58:59.884 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:59.884 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:59.884 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:59.884 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:59.885 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:58:59.885 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:58:59.885 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:59.885 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:59.886 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:58:59.886 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:58:59.886 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:58:59.886 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:58:59.887 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:59.887 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:58:59.887 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:58:59.887 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:58:59.888 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:58:59.888 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:58:59.888 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:58:59.888 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:58:59.889 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:58:59.889 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:58:59.890 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:59.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:59.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:59.992 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:59.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:59.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:59.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:59.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:59.993 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:00.353 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:00.353 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:00.354 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:00.354 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:00.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:00.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:00.843 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:00.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:00.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:00.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:00.844 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:00.844 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:00.845 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:01.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989060 2021-05-12T12:59:01.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989060 '!=' null 2021-05-12T12:59:01.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:01.329 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:01.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:01.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:01.421 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:01.422 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:01.422 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:01.422 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:01.422 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:01.423 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:01.423 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:01.423 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:01.423 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:01.424 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:01.424 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:01.424 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:01.424 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:01.424 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:01.425 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:01.425 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:01.425 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:01.426 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:01.426 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:01.426 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:01.426 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:01.427 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:01.427 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:01.427 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:01.427 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:01.427 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:01.428 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:01.428 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:01.428 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:01.428 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:01.429 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:01.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:01.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:01.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:01.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:01.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:01.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:01.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:01.533 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:01.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:01.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:01.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:01.906 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:02.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:02.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:02.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:02.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:02.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:02.396 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:02.397 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:02.397 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:02.397 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:02.877 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368170 2021-05-12T12:59:02.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368170 '!=' null 2021-05-12T12:59:02.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:02.878 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:02.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:02.879 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:02.971 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:02.971 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:02.971 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:02.972 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:02.972 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:02.972 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:02.972 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:02.972 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:02.973 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:02.973 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:02.973 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:02.973 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:02.973 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:02.974 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:02.974 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:02.974 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:02.974 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:02.975 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:02.975 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:02.975 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:02.975 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:02.975 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:02.976 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:02.976 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:02.976 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:02.976 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:02.977 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:02.977 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:02.977 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:02.977 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:02.979 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:03.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:03.081 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:03.081 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:03.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:03.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:03.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:03.082 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:03.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:03.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:03.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:03.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:03.448 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:04.120 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:04.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:04.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:04.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:04.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:04.121 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:04.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:04.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:04.122 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:04.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989127 2021-05-12T12:59:04.604 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989127 '!=' null 2021-05-12T12:59:04.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:04.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:04.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:04.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:04.698 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:04.699 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:04.699 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:04.699 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:04.699 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:04.700 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:04.700 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:04.700 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:04.700 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:04.701 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:04.701 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:04.701 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:04.701 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:04.701 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:04.702 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:04.702 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:04.702 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:04.702 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:04.703 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:04.703 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:04.703 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:04.703 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:04.704 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:04.704 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:04.704 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:04.704 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:04.704 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:04.705 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:04.705 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:04.705 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:04.707 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:04.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:04.809 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:04.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:04.809 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:04.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:04.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:04.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:04.810 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:05.165 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:05.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:05.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:05.166 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:05.665 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:05.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:05.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:05.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:05.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:05.666 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:05.667 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:05.667 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:05.667 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:06.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368208 2021-05-12T12:59:06.156 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368208 '!=' null 2021-05-12T12:59:06.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:06.157 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:06.157 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:06.157 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:06.250 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:06.251 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:06.251 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:06.251 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:06.251 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:06.252 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:06.252 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:06.252 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:06.252 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:06.252 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:06.253 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:06.253 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:06.253 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:06.253 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:06.253 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:06.254 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:06.254 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:06.254 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:06.255 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:06.255 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:06.255 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:06.255 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:06.255 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:06.256 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:06.256 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:06.256 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:06.256 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:06.257 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:06.257 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:06.257 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:06.259 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:06.360 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:06.361 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:06.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:06.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:06.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:06.361 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:06.362 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:06.362 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:06.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:06.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:06.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:06.714 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:07.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:07.198 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:07.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:07.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:07.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:07.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:07.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:07.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:07.200 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:07.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368263 2021-05-12T12:59:07.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368263 '!=' null 2021-05-12T12:59:07.690 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:07.691 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:07.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:07.691 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:07.783 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:07.784 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:07.784 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:07.784 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:07.784 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:07.785 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:07.785 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:07.785 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:07.785 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:07.785 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:07.786 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:07.786 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:07.786 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:07.786 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:07.787 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:07.787 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:07.787 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:07.787 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:07.787 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:07.788 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:07.788 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:07.788 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:07.789 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:07.789 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:07.789 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:07.789 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:07.789 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:07.790 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:07.790 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:07.790 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:07.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:07.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:07.893 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:07.894 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:07.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:07.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:07.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:07.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:07.895 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:08.255 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:08.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:08.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:08.256 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:08.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:08.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:08.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:08.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:08.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:08.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:08.750 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:08.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:08.750 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:09.253 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368263 2021-05-12T12:59:09.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368263 '!=' null 2021-05-12T12:59:09.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:09.254 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:09.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:09.255 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:09.347 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:09.348 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:09.348 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:09.348 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:09.348 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:09.349 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:09.349 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:09.349 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:09.349 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:09.350 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:09.350 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:09.350 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:09.350 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:09.351 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:09.351 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:09.351 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:09.351 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:09.352 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:09.352 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:09.352 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:09.352 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:09.353 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:09.353 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:09.353 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:09.353 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:09.353 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:09.354 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:09.354 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:09.354 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:09.354 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:09.356 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:09.457 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:09.458 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:09.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:09.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:09.458 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:09.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:09.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:09.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:09.813 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:09.813 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:09.814 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:09.814 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:10.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:10.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:10.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:10.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:10.299 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:10.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:10.300 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:10.300 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:10.300 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:10.833 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989285 2021-05-12T12:59:10.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989285 '!=' null 2021-05-12T12:59:10.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:10.834 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:10.835 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:10.835 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:10.927 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:10.928 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:10.928 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:10.928 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:10.928 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:10.929 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:10.929 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:10.929 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:10.929 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:10.930 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:10.930 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:10.930 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:10.930 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:10.931 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:10.931 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:10.931 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:10.931 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:10.932 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:10.932 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:10.932 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:10.932 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:10.932 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:10.933 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:10.933 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:10.933 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:10.933 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:10.933 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:10.934 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:10.934 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:10.934 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:10.935 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:11.037 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:11.038 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:11.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:11.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:11.038 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:11.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:11.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:11.039 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:11.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:11.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:11.398 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:11.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:11.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:11.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:11.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:11.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:11.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:11.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:11.887 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:11.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:11.888 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:12.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241941 2021-05-12T12:59:12.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241941 '!=' null 2021-05-12T12:59:12.368 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:12.369 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:12.369 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:12.369 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:12.464 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:12.464 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:12.465 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:12.465 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:12.465 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:12.466 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:12.466 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:12.466 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:12.467 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:12.467 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:12.467 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:12.467 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:12.467 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:12.468 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:12.468 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:12.468 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:12.468 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:12.469 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:12.469 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:12.469 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:12.469 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:12.469 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:12.470 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:12.470 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:12.470 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:12.470 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:12.470 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:12.471 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:12.471 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:12.471 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:12.472 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:12.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:12.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:12.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:12.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:12.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:12.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:12.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:12.576 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:12.931 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:12.932 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:12.932 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:12.933 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:13.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:13.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:13.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:13.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:13.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:13.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:13.409 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:13.410 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:13.410 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:13.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862839 2021-05-12T12:59:13.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862839 '!=' null 2021-05-12T12:59:13.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:13.891 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:13.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:13.892 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:13.984 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:13.984 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:13.984 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:13.985 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:13.985 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:13.985 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:13.985 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:13.985 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:13.986 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:13.986 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:13.986 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:13.986 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:13.986 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:13.987 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:13.987 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:13.987 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:13.987 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:13.988 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:13.988 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:13.988 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:13.988 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:13.988 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:13.989 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:13.989 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:13.989 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:13.989 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:13.989 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:13.990 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:13.990 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:13.990 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:13.992 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:14.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:14.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:14.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:14.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:14.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:14.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:14.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:14.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:14.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:14.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:14.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:14.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:14.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:14.945 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:14.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:14.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:14.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:14.946 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:14.947 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:14.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:14.947 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:15.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862839 2021-05-12T12:59:15.436 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862839 '!=' null 2021-05-12T12:59:15.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:15.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:15.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:15.437 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:15.531 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:15.532 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:15.532 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:15.532 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:15.532 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:15.532 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:15.533 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:15.533 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:15.533 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:15.533 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:15.534 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:15.534 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:15.534 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:15.534 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:15.535 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:15.535 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:15.535 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:15.535 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:15.536 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:15.536 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:15.536 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:15.537 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:15.537 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:15.537 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:15.537 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:15.538 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:15.538 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:15.538 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:15.538 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:15.539 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:15.539 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:15.641 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:15.642 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:15.642 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:15.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:15.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:15.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:15.643 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:15.644 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:15.999 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:16.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:16.000 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:16.001 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:16.481 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:16.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:16.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:16.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:16.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:16.482 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:16.483 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:16.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:16.483 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:16.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241866 2021-05-12T12:59:16.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241866 '!=' null 2021-05-12T12:59:16.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:16.997 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:16.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:16.998 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:17.090 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:17.090 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:17.090 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:17.091 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:17.091 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:17.091 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:17.091 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:17.092 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:17.092 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:17.092 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:17.092 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:17.092 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:17.093 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:17.093 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:17.093 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:17.093 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:17.093 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:17.094 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:17.094 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:17.094 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:17.094 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:17.095 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:17.095 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:17.095 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:17.095 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:17.095 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:17.096 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:17.096 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:17.096 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:17.097 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:17.098 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:17.199 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:17.200 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:17.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:17.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:17.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:17.201 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:17.202 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:17.202 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:17.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:17.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:17.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:17.568 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:18.057 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:18.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:18.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:18.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:18.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:18.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:18.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:18.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:18.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:18.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241869 2021-05-12T12:59:18.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241869 '!=' null 2021-05-12T12:59:18.548 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:18.549 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:18.549 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:18.549 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:18.642 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:18.643 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:18.643 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:18.643 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:18.643 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:18.644 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:18.644 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:18.644 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:18.644 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:18.644 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:18.645 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:18.645 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:18.645 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:18.645 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:18.646 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:18.646 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:18.646 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:18.646 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:18.646 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:18.647 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:18.647 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:18.647 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:18.647 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:18.648 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:18.648 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:18.648 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:18.648 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:18.649 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:18.649 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:18.649 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:18.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:18.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:18.753 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:18.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:18.754 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:18.754 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:18.754 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:18.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:18.755 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:19.109 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:19.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:19.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:19.110 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:19.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:19.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:19.605 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:19.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:19.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:19.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:19.606 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:19.606 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:19.607 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:20.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862738 2021-05-12T12:59:20.094 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862738 '!=' null 2021-05-12T12:59:20.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:20.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:20.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:20.095 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:20.188 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:20.188 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:20.189 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:20.189 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:20.189 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:20.189 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:20.190 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:20.190 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:20.190 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:20.190 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:20.190 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:20.191 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:20.191 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:20.191 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:20.191 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:20.192 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:20.192 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:20.192 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:20.192 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:20.193 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:20.193 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:20.193 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:20.193 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:20.193 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:20.194 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:20.194 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:20.194 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:20.194 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:20.195 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:20.195 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:20.196 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:20.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:20.298 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:20.298 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:20.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:20.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:20.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:20.299 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:20.300 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:20.650 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:20.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:20.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:20.651 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:21.172 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:21.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:21.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:21.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:21.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:21.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:21.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:21.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:21.174 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:21.668 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241830 2021-05-12T12:59:21.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241830 '!=' null 2021-05-12T12:59:21.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:21.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:21.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:21.669 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:21.762 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:21.763 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:21.763 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:21.763 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:21.764 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:21.764 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:21.764 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:21.764 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:21.765 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:21.765 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:21.765 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:21.765 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:21.766 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:21.766 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:21.766 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:21.766 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:21.767 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:21.767 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:21.767 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:21.767 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:21.767 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:21.768 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:21.768 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:21.768 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:21.768 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:21.768 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:21.769 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:21.769 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:21.769 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:21.769 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:21.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:21.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:21.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:21.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:21.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:21.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:21.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:21.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:21.874 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:22.233 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:22.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:22.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:22.234 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:22.715 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:22.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:22.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:22.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:22.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:22.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:22.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:22.717 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:22.717 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:23.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241830 2021-05-12T12:59:23.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241830 '!=' null 2021-05-12T12:59:23.208 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:23.209 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:23.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:23.209 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:23.303 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:23.303 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:23.303 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:23.303 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:23.304 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:23.304 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:23.304 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:23.304 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:23.305 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:23.305 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:23.305 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:23.305 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:23.306 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:23.306 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:23.306 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:23.306 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:23.307 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:23.307 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:23.307 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:23.307 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:23.307 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:23.308 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:23.308 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:23.308 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:23.308 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:23.308 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:23.309 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:23.309 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:23.309 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:23.309 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:23.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:23.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:23.412 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:23.412 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:23.412 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:23.413 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:23.413 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:23.413 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:23.413 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:23.773 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:23.773 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:23.774 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:23.774 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:24.263 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:24.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:24.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:24.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:24.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:24.265 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:24.266 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:24.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:24.266 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:24.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862756 2021-05-12T12:59:24.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862756 '!=' null 2021-05-12T12:59:24.748 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:24.749 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:24.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:24.749 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:24.840 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:24.841 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:24.841 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:24.841 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:24.842 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:24.842 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:24.842 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:24.842 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:24.843 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:24.843 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:24.843 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:24.844 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:24.844 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:24.844 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:24.845 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:24.845 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:24.845 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:24.846 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:24.846 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:24.846 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:24.847 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:24.847 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:24.847 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:24.848 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:24.848 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:24.848 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:24.849 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:24.849 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:24.849 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:24.849 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:24.850 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:24.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:24.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:24.951 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:24.951 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:24.951 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:24.952 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:24.952 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:24.952 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:25.309 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:25.309 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:25.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:25.310 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:25.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:25.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:25.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:25.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:25.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:25.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:25.816 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:25.817 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:25.817 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:26.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241858 2021-05-12T12:59:26.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241858 '!=' null 2021-05-12T12:59:26.306 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:26.307 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:26.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:26.307 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:26.399 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:26.399 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:26.399 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:26.400 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:26.400 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:26.400 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:26.400 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:26.401 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:26.401 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:26.401 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:26.401 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:26.401 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:26.402 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:26.402 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:26.402 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:26.402 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:26.403 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:26.403 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:26.403 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:26.403 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:26.404 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:26.404 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:26.404 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:26.404 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:26.404 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:26.405 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:26.405 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:26.405 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:26.405 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:26.405 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:26.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:26.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:26.509 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:26.509 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:26.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:26.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:26.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:26.510 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:26.511 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:26.872 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:26.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:26.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:26.873 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:27.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:27.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:27.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:27.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:27.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:27.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:27.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:27.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:27.375 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:27.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241869 2021-05-12T12:59:27.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241869 '!=' null 2021-05-12T12:59:27.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:27.861 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:27.861 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:27.861 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:27.957 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:27.957 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:27.957 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:27.958 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:27.958 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:27.958 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:27.958 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:27.959 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:27.959 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:27.959 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:27.959 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:27.959 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:27.960 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:27.960 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:27.960 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:27.960 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:27.961 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:27.961 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:27.961 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:27.961 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:27.961 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:27.962 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:27.962 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:27.962 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:27.962 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:27.962 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:27.963 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:27.963 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:27.963 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:27.963 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:27.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:28.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:28.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:28.067 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:28.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:28.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:28.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:28.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:28.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:28.430 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:28.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:28.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:28.431 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:28.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:28.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:28.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:28.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:28.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:28.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:28.907 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:28.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:28.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:29.398 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241869 2021-05-12T12:59:29.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241869 '!=' null 2021-05-12T12:59:29.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:29.399 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:29.399 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:29.400 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:29.492 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:29.492 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:29.492 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:29.493 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:29.493 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:29.493 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:29.493 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:29.494 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:29.494 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:29.494 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:29.494 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:29.495 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:29.495 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:29.495 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:29.495 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:29.496 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:29.496 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:29.496 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:29.496 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:29.496 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:29.497 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:29.497 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:29.497 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:29.497 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:29.498 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:29.498 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:29.498 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:29.498 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:29.498 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:29.499 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:29.501 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:29.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:29.602 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:29.603 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:29.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:29.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:29.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:29.604 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:29.605 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:29.974 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:29.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:29.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:29.975 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:30.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:30.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:30.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:30.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:30.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:30.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:30.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:30.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:30.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:30.966 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862734 2021-05-12T12:59:30.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862734 '!=' null 2021-05-12T12:59:30.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:30.967 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:30.967 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:30.968 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:31.060 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:31.060 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:31.060 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:31.061 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:31.061 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:31.061 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:31.061 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:31.061 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:31.062 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:31.062 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:31.062 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:31.062 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:31.063 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:31.063 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:31.063 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:31.063 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:31.063 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:31.064 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:31.064 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:31.064 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:31.064 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:31.065 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:31.065 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:31.065 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:31.065 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:31.065 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:31.066 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:31.066 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:31.066 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:31.066 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:31.068 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:31.170 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:31.285 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:31.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:31.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:31.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:31.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:31.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:31.287 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:31.525 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:31.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:31.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:31.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:32.437 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:32.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:32.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:32.438 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:32.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:32.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:32.439 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:32.439 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:32.440 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:32.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241823 2021-05-12T12:59:32.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241823 '!=' null 2021-05-12T12:59:32.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:32.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:32.922 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:32.922 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:33.014 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:33.015 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:33.015 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:33.015 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:33.015 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:33.015 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:33.016 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:33.016 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:33.016 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:33.016 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:33.017 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:33.017 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:33.017 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:33.017 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:33.017 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:33.018 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:33.018 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:33.018 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:33.018 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:33.019 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:33.019 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:33.019 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:33.019 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:33.020 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:33.020 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:33.020 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:33.020 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:33.020 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:33.021 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:33.021 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:33.023 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:33.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:33.125 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:33.126 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:33.126 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:33.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:33.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:33.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:33.127 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:33.485 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:33.485 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:33.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:33.486 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:33.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:33.968 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:33.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:33.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:33.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:33.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:33.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:33.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:33.970 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:34.468 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989083 2021-05-12T12:59:34.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989083 '!=' null 2021-05-12T12:59:34.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:34.469 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:34.469 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:34.470 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:34.562 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:34.563 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:34.563 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:34.563 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:34.563 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:34.564 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:34.564 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:34.564 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:34.564 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:34.565 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:34.565 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:34.565 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:34.565 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:34.566 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:34.566 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:34.566 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:34.566 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:34.566 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:34.567 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:34.567 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:34.567 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:34.567 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:34.568 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:34.568 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:34.568 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:34.568 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:34.568 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:34.569 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:34.569 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:34.569 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:34.571 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:34.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:34.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:34.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:34.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:34.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:34.674 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:34.674 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:34.674 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:35.033 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:35.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:35.033 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:35.034 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:35.530 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:35.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:35.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:35.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:35.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:35.531 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:35.532 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:35.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:35.532 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:36.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368179 2021-05-12T12:59:36.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368179 '!=' null 2021-05-12T12:59:36.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:36.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:36.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:36.011 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:36.105 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:36.105 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:36.105 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:36.106 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:36.106 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:36.106 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:36.106 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:36.107 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:36.107 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:36.107 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:36.107 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:36.107 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:36.108 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:36.108 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:36.108 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:36.108 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:36.108 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:36.109 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:36.109 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:36.109 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:36.109 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:36.109 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:36.110 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:36.110 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:36.110 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:36.111 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:36.111 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:36.111 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:36.111 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:36.111 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:36.113 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:36.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:36.216 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:36.216 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:36.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:36.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:36.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:36.217 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:36.218 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:36.574 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:36.574 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:36.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:36.575 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:37.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:37.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:37.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:37.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:37.062 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:37.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:37.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:37.063 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:37.063 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:37.546 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368220 2021-05-12T12:59:37.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368220 '!=' null 2021-05-12T12:59:37.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:37.547 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:37.547 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:37.548 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:37.640 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:37.641 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:37.641 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:37.641 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:37.641 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:37.642 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:37.642 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:37.642 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:37.642 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:37.643 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:37.643 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:37.643 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:37.643 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:37.644 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:37.644 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:37.644 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:37.644 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:37.644 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:37.645 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:37.645 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:37.645 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:37.645 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:37.645 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:37.646 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:37.646 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:37.646 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:37.646 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:37.647 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:37.647 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:37.647 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:37.649 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:37.751 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:37.751 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:37.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:37.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:37.752 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:37.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:37.753 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:38.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:38.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:38.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:38.104 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:38.588 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:38.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:38.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:38.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:38.589 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:38.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:38.590 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:38.590 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:38.590 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:39.078 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368220 2021-05-12T12:59:39.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368220 '!=' null 2021-05-12T12:59:39.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:39.079 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:39.079 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:39.080 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:39.172 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:39.173 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:39.173 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:39.173 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:39.173 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:39.174 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:39.174 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:39.174 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:39.174 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:39.174 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:39.175 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:39.175 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:39.175 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:39.175 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:39.175 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:39.176 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:39.176 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:39.176 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:39.176 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:39.177 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:39.177 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:39.177 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:39.177 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:39.178 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:39.178 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:39.178 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:39.178 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:39.179 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:39.179 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:39.179 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:39.181 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:39.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:39.284 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:39.284 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:39.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:39.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:39.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:39.285 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:39.286 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:39.646 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:39.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:39.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:39.647 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:40.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:40.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:40.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:40.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:40.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:40.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:40.136 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:40.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:40.136 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:40.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862798 2021-05-12T12:59:40.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862798 '!=' null 2021-05-12T12:59:40.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:40.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:40.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:40.618 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:40.710 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:40.711 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:40.711 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:40.711 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:40.712 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:40.712 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:40.712 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:40.712 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:40.713 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:40.713 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:40.713 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:40.713 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:40.713 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:40.714 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:40.714 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:40.714 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:40.714 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:40.715 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:40.715 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:40.715 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:40.715 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:40.716 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:40.716 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:40.716 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:40.716 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:40.716 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:40.717 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:40.717 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:40.717 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:40.717 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:40.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:40.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:40.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:40.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:40.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:40.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:40.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:40.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:40.823 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:41.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:41.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:41.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:41.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:41.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:41.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:41.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:41.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:41.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:41.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:41.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:41.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:41.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:42.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241862 2021-05-12T12:59:42.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241862 '!=' null 2021-05-12T12:59:42.166 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:42.167 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:42.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:42.167 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:42.260 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:42.260 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:42.260 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:42.261 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:42.261 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:42.261 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:42.261 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:42.262 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:42.262 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:42.262 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:42.262 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:42.262 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:42.263 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:42.263 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:42.263 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:42.263 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:42.263 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:42.264 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:42.264 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:42.264 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:42.264 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:42.265 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:42.265 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:42.265 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:42.265 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:42.265 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:42.266 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:42.266 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:42.266 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:42.266 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:42.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:42.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:42.370 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:42.371 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:42.371 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:42.371 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:42.371 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:42.371 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:42.372 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:42.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:42.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:42.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:42.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:43.219 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:43.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:43.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:43.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:43.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:43.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:43.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:43.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:43.221 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:43.711 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989231 2021-05-12T12:59:43.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989231 '!=' null 2021-05-12T12:59:43.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:43.712 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:43.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:43.713 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:43.806 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:43.806 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:43.806 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:43.806 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:43.807 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:43.807 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:43.807 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:43.807 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:43.807 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:43.808 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:43.808 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:43.808 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:43.808 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:43.808 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:43.809 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:43.809 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:43.809 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:43.809 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:43.810 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:43.810 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:43.810 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:43.810 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:43.810 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:43.811 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:43.811 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:43.811 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:43.811 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:43.811 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:43.812 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:43.812 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:43.814 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:43.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:43.916 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:43.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:43.916 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:43.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:43.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:43.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:43.917 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:44.275 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:44.275 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:44.275 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:44.276 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:44.762 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:44.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:44.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:44.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:44.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:44.763 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:44.764 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:44.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:44.764 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:45.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989231 2021-05-12T12:59:45.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989231 '!=' null 2021-05-12T12:59:45.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:45.248 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:45.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:45.249 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:45.343 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:45.344 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:45.344 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:45.344 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:45.344 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:45.345 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:45.345 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:45.345 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:45.345 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:45.346 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:45.346 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:45.346 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:45.346 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:45.346 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:45.347 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:45.347 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:45.347 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:45.347 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:45.348 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:45.348 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:45.348 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:45.348 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:45.348 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:45.349 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:45.349 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:45.349 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:45.349 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:45.350 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:45.350 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:45.350 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:45.351 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:45.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:45.453 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:45.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:45.454 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:45.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:45.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:45.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:45.455 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:45.815 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:45.815 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:45.816 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:45.816 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:46.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:46.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:46.310 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:46.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:46.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:46.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:46.311 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:46.311 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:46.312 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:46.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241933 2021-05-12T12:59:46.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241933 '!=' null 2021-05-12T12:59:46.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:46.801 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:46.802 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:46.802 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:46.893 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:46.893 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:46.894 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:46.894 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:46.894 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:46.894 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:46.895 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:46.895 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:46.895 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:46.895 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:46.896 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:46.896 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:46.896 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:46.896 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:46.897 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:46.897 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:46.897 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:46.897 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:46.897 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:46.898 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:46.898 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:46.898 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:46.899 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:46.899 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:46.899 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:46.899 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:46.899 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:46.900 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:46.900 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:46.900 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:46.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:47.003 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:47.004 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:47.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:47.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:47.004 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:47.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:47.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:47.005 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:47.364 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:47.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:47.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:47.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:47.840 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:47.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:47.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:47.841 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:47.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:47.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:47.842 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:47.842 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:47.843 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:48.318 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241954 2021-05-12T12:59:48.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241954 '!=' null 2021-05-12T12:59:48.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:48.319 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:48.319 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:48.320 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:48.412 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:48.413 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:48.413 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:48.413 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:48.414 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:48.414 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:48.414 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:48.414 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:48.415 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:48.415 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:48.415 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:48.415 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:48.416 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:48.416 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:48.416 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:48.416 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:48.416 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:48.417 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:48.417 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:48.417 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:48.417 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:48.417 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:48.418 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:48.418 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:48.418 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:48.418 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:48.419 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:48.419 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:48.419 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:48.419 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:48.421 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:48.522 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:48.523 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:48.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:48.523 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:48.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:48.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:48.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:48.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:48.880 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:48.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:48.880 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:48.881 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:49.374 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:49.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:49.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:49.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:49.375 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:49.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:49.376 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:49.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:49.376 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:49.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862919 2021-05-12T12:59:49.855 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862919 '!=' null 2021-05-12T12:59:49.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:49.856 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:49.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:49.856 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:49.948 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:49.949 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:49.949 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:49.949 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:49.949 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:49.949 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:49.950 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:49.950 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:49.950 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:49.950 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:49.951 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:49.951 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:49.951 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:49.951 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:49.951 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:49.952 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:49.952 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:49.952 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:49.953 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:49.953 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:49.953 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:49.953 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:49.953 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:49.954 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:49.954 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:49.954 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:49.954 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:49.955 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:49.955 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:49.955 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:49.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:50.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:50.059 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:50.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:50.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:50.059 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:50.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:50.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:50.060 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:50.411 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:50.412 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:50.412 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:50.413 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:50.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:50.901 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:50.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:50.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:50.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:50.902 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:50.903 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:50.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:50.903 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:51.382 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862919 2021-05-12T12:59:51.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862919 '!=' null 2021-05-12T12:59:51.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:51.383 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:51.383 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:51.384 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:51.476 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:51.477 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:51.477 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:51.477 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:51.477 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:51.478 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:51.478 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:51.478 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:51.478 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:51.479 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:51.479 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:51.479 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:51.479 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:51.479 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:51.480 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:51.480 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:51.480 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:51.480 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:51.480 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:51.481 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:51.481 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:51.481 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:51.481 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:51.481 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:51.482 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:51.482 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:51.482 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:51.482 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:51.483 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:51.483 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:51.484 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:51.586 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:51.587 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:51.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:51.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:51.587 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:51.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:51.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:51.588 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:51.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:51.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:51.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:51.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:52.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:52.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:52.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:52.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:52.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:52.446 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:52.447 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:52.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:52.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:52.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368228 2021-05-12T12:59:52.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368228 '!=' null 2021-05-12T12:59:52.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:52.959 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:52.959 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:52.959 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:53.052 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:53.052 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:53.053 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:53.053 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:53.053 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:53.053 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:53.053 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:53.054 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:53.054 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:53.055 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:53.056 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:53.056 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:53.056 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:53.056 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:53.057 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:53.057 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:53.057 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:53.058 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:53.058 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:53.058 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:53.058 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:53.059 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:53.059 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:53.059 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:53.059 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:53.060 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:53.060 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:53.060 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:53.060 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:53.060 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:53.061 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:53.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:53.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:53.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:53.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:53.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:53.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:53.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:53.164 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:53.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:53.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:53.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:53.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:54.045 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:54.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:54.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:54.046 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:54.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:54.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:54.047 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:54.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:54.047 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:54.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989174 2021-05-12T12:59:54.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989174 '!=' null 2021-05-12T12:59:54.533 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:54.534 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:54.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:54.534 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:54.627 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:54.627 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:54.628 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:54.628 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:54.628 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:54.628 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:54.629 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:54.629 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:54.629 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:54.629 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:54.630 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:54.630 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:54.630 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:54.630 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:54.631 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:54.631 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:54.631 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:54.632 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:54.632 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:54.632 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:54.632 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:54.633 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:54.633 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:54.633 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:54.633 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:54.634 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:54.634 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:54.634 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:54.634 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:54.635 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:54.635 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:54.737 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:54.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:54.738 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:54.738 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:54.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:54.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:54.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:54.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:55.095 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:55.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:55.096 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:55.097 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:55.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:55.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:55.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:55.609 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:55.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:55.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:55.610 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:55.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:55.610 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:56.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241874 2021-05-12T12:59:56.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241874 '!=' null 2021-05-12T12:59:56.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:56.092 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:56.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:56.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:56.185 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:56.185 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:56.186 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:56.186 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:56.186 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:56.186 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:56.187 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:56.187 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:56.187 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:56.187 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:56.187 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:56.188 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:56.188 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:56.188 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:56.188 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:56.188 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:56.189 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:56.189 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:56.189 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:56.189 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:56.190 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:56.190 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:56.190 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:56.190 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:56.191 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:56.191 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:56.191 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:56.191 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:56.191 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:56.192 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:56.193 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:56.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:56.295 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:56.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:56.295 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:56.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:56.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:56.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:56.296 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:56.654 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:56.654 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:56.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:56.655 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:57.133 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:57.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:57.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:57.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:57.134 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:57.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:57.135 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:57.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:57.135 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:57.617 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241871 2021-05-12T12:59:57.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241871 '!=' null 2021-05-12T12:59:57.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:57.618 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:57.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:57.619 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:57.710 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:57.711 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:57.711 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:57.711 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:57.712 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:57.712 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:57.712 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:57.712 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:57.712 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:57.713 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:57.713 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:57.713 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:57.713 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:57.714 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:57.714 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:57.714 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:57.714 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:57.714 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:57.715 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:57.715 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:57.715 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:57.715 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:57.716 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:57.716 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:57.716 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:57.716 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:57.716 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:57.717 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:57.717 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:57.717 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:57.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:57.820 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:57.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:57.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:57.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:57.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:57.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:57.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:57.822 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:58.179 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:58.179 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:58.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:58.180 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:58.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:58.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:58.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:58.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:58.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:58.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:58.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:58.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:58.673 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:59.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241871 2021-05-12T12:59:59.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241871 '!=' null 2021-05-12T12:59:59.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:59.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:59.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:59.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:59.256 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T12:59:59.256 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T12:59:59.257 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:59.257 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:59.258 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:59.258 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T12:59:59.258 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:59.259 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T12:59:59.259 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:59.260 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:59.260 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:59.260 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:59.261 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T12:59:59.261 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T12:59:59.262 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:59.262 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:59.263 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T12:59:59.263 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T12:59:59.264 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T12:59:59.264 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T12:59:59.265 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:59.265 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T12:59:59.265 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T12:59:59.266 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T12:59:59.266 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T12:59:59.267 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T12:59:59.267 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T12:59:59.268 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T12:59:59.268 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T12:59:59.268 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T12:59:59.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:59.366 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:59.367 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:59.367 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:59.368 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:59.368 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:59.369 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:59.369 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:59.369 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:59.728 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:59.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:59.728 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:59.729 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:00.211 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:00.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:00.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:00.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:00.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:00.212 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:00.213 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:00.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:00.213 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:00.702 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862689 2021-05-12T13:00:00.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862689 '!=' null 2021-05-12T13:00:00.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:00.703 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:00.704 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:00.704 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:00.796 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:00.796 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:00.797 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:00.797 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:00.797 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:00.797 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:00.798 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:00.798 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:00.798 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:00.798 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:00.799 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:00.799 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:00.799 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:00.799 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:00.800 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:00.800 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:00.800 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:00.800 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:00.800 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:00.801 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:00.801 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:00.801 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:00.801 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:00.802 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:00.802 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:00.802 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:00.802 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:00.803 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:00.803 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:00.803 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:00.804 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:00.905 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:00.906 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:00.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:00.907 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:00.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:00.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:00.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:00.908 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:01.264 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:01.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:01.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:01.265 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:01.767 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:01.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:01.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:01.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:01.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:01.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:01.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:01.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:01.769 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:02.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241743 2021-05-12T13:00:02.241 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241743 '!=' null 2021-05-12T13:00:02.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:02.242 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:02.242 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:02.242 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:02.335 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:02.336 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:02.336 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:02.336 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:02.336 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:02.337 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:02.337 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:02.337 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:02.337 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:02.338 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:02.338 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:02.338 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:02.338 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:02.338 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:02.339 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:02.339 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:02.339 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:02.339 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:02.340 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:02.340 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:02.340 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:02.340 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:02.340 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:02.341 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:02.341 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:02.341 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:02.341 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:02.342 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:02.342 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:02.342 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:02.343 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:02.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:02.445 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:02.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:02.446 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:02.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:02.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:02.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:02.447 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:02.828 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:02.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:02.828 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:02.829 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:03.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:03.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:03.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:03.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:03.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:03.314 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:03.315 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:03.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:03.315 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:03.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862675 2021-05-12T13:00:03.791 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862675 '!=' null 2021-05-12T13:00:03.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:03.792 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:03.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:03.792 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:03.883 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:03.884 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:03.884 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:03.884 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:03.884 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:03.885 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:03.885 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:03.885 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:03.885 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:03.886 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:03.886 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:03.886 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:03.886 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:03.887 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:03.887 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:03.887 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:03.887 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:03.888 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:03.888 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:03.888 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:03.888 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:03.889 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:03.889 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:03.889 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:03.889 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:03.890 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:03.890 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:03.890 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:03.890 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:03.890 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:03.892 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:03.993 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:03.994 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:03.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:03.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:03.994 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:03.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:03.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:03.995 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:04.354 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:04.354 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:04.354 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:04.355 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:04.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:04.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:04.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:04.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:04.859 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:04.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:04.860 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:04.860 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:04.860 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:05.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862675 2021-05-12T13:00:05.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862675 '!=' null 2021-05-12T13:00:05.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:05.535 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:05.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:05.536 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:05.628 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:05.629 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:05.629 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:05.629 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:05.629 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:05.630 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:05.630 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:05.630 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:05.630 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:05.631 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:05.631 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:05.631 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:05.631 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:05.631 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:05.632 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:05.632 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:05.632 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:05.632 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:05.633 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:05.633 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:05.633 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:05.633 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:05.634 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:05.634 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:05.634 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:05.634 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:05.634 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:05.635 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:05.635 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:05.635 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:05.636 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:05.738 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:05.739 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:05.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:05.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:05.739 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:05.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:05.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:05.740 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:06.102 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:06.102 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:06.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:06.103 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:06.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:06.582 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:06.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:06.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:06.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:06.583 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:06.584 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:06.584 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:06.584 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:07.063 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241837 2021-05-12T13:00:07.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241837 '!=' null 2021-05-12T13:00:07.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:07.064 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:07.064 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:07.065 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:07.158 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:07.159 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:07.159 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:07.160 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:07.160 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:07.161 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:07.161 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:07.161 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:07.161 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:07.162 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:07.162 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:07.162 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:07.163 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:07.163 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:07.163 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:07.163 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:07.164 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:07.164 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:07.164 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:07.164 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:07.164 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:07.165 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:07.165 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:07.165 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:07.166 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:07.166 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:07.166 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:07.167 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:07.167 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:07.167 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:07.168 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:07.268 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:07.269 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:07.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:07.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:07.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:07.269 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:07.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:07.270 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:07.633 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:07.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:07.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:07.634 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:08.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:08.128 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:08.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:08.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:08.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:08.129 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:08.130 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:08.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:08.130 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:08.614 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241835 2021-05-12T13:00:08.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241835 '!=' null 2021-05-12T13:00:08.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:08.615 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:08.615 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:08.616 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:08.708 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:08.708 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:08.709 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:08.709 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:08.709 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:08.709 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:08.710 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:08.710 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:08.710 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:08.710 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:08.711 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:08.711 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:08.711 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:08.711 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:08.711 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:08.712 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:08.712 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:08.712 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:08.712 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:08.713 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:08.713 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:08.713 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:08.713 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:08.713 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:08.714 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:08.714 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:08.714 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:08.714 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:08.715 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:08.715 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:08.716 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:08.818 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:08.819 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:08.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:08.819 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:08.820 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:08.820 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:08.820 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:08.821 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:09.183 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:09.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:09.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:09.183 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:09.669 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:09.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:09.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:09.670 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:09.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:09.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:09.671 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:09.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:09.671 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:10.153 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862741 2021-05-12T13:00:10.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862741 '!=' null 2021-05-12T13:00:10.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:10.154 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:10.155 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:10.155 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:10.248 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:10.248 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:10.248 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:10.249 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:10.249 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:10.249 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:10.249 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:10.250 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:10.250 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:10.250 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:10.250 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:10.251 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:10.251 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:10.251 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:10.251 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:10.252 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:10.252 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:10.252 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:10.252 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:10.253 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:10.253 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:10.253 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:10.254 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:10.254 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:10.254 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:10.254 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:10.254 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:10.255 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:10.255 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:10.255 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:10.256 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:10.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:10.358 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:10.358 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:10.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:10.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:10.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:10.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:10.359 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:10.717 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:10.718 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:10.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:10.719 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:11.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:11.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:11.205 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:11.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:11.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:11.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:11.206 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:11.206 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:11.207 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:11.745 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241888 2021-05-12T13:00:11.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241888 '!=' null 2021-05-12T13:00:11.746 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:11.747 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:11.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:11.747 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:11.840 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:11.840 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:11.841 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:11.841 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:11.841 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:11.842 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:11.842 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:11.842 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:11.843 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:11.843 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:11.843 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:11.843 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:11.844 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:11.844 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:11.844 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:11.844 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:11.845 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:11.845 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:11.845 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:11.845 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:11.845 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:11.846 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:11.846 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:11.846 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:11.846 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:11.847 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:11.847 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:11.847 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:11.847 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:11.848 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:11.848 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:11.950 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:11.951 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:11.951 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:11.951 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:11.951 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:11.952 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:11.952 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:11.952 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:12.313 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:12.314 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:12.314 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:12.314 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:12.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:12.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:12.799 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:12.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:12.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:12.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:12.800 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:12.800 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:12.801 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:13.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241888 2021-05-12T13:00:13.292 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241888 '!=' null 2021-05-12T13:00:13.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:13.293 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:13.293 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:13.293 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:13.386 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:13.387 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:13.387 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:13.387 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:13.388 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:13.388 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:13.388 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:13.388 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:13.389 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:13.389 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:13.389 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:13.389 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:13.389 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:13.390 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:13.390 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:13.390 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:13.390 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:13.391 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:13.391 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:13.391 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:13.391 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:13.392 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:13.392 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:13.392 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:13.392 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:13.392 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:13.393 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:13.393 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:13.393 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:13.393 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:13.395 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:13.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:13.497 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:13.497 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:13.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:13.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:13.498 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:13.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:13.499 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:13.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:13.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:13.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:13.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:14.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:14.345 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:14.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:14.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:14.346 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:14.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:14.347 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:14.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:14.347 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:14.867 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862881 2021-05-12T13:00:14.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862881 '!=' null 2021-05-12T13:00:14.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:14.868 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:14.868 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:14.869 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:14.961 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:14.961 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:14.961 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:14.962 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:14.962 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:14.962 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:14.962 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:14.962 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:14.963 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:14.963 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:14.963 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:14.963 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:14.964 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:14.964 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:14.964 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:14.964 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:14.964 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:14.965 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:14.965 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:14.965 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:14.965 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:14.965 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:14.966 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:14.966 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:14.966 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:14.966 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:14.967 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:14.967 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:14.967 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:14.967 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:14.969 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:15.070 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:15.071 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:15.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:15.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:15.071 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:15.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:15.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:15.072 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:15.428 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:15.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:15.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:15.429 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:15.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:15.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:15.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:15.920 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:15.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:15.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:15.921 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:15.921 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:15.922 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:16.407 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241927 2021-05-12T13:00:16.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241927 '!=' null 2021-05-12T13:00:16.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:16.408 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:16.408 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:16.409 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:16.502 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:16.502 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:16.503 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:16.503 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:16.503 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:16.503 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:16.504 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:16.504 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:16.504 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:16.504 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:16.504 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:16.505 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:16.505 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:16.505 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:16.505 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:16.506 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:16.506 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:16.506 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:16.507 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:16.507 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:16.507 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:16.507 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:16.507 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:16.508 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:16.508 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:16.508 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:16.508 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:16.508 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:16.509 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:16.509 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:16.510 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:16.612 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:16.613 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:16.613 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:16.613 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:16.613 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:16.613 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:16.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:16.614 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:16.975 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:16.976 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:16.976 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:16.977 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:17.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:17.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:17.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:17.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:17.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:17.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:17.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:17.467 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:17.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:17.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241930 2021-05-12T13:00:17.957 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241930 '!=' null 2021-05-12T13:00:17.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:17.958 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:17.958 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:17.959 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:18.050 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:18.050 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:18.050 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:18.051 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:18.051 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:18.051 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:18.051 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:18.052 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:18.052 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:18.052 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:18.053 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:18.053 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:18.053 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:18.053 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:18.054 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:18.054 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:18.054 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:18.054 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:18.055 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:18.055 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:18.055 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:18.056 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:18.056 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:18.056 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:18.056 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:18.057 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:18.057 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:18.057 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:18.057 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:18.058 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:18.058 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:18.160 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:18.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:18.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:18.161 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:18.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:18.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:18.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:18.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:18.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:18.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:18.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:18.526 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:19.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:19.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:19.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:19.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:19.011 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:19.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:19.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:19.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:19.012 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:19.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862920 2021-05-12T13:00:19.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862920 '!=' null 2021-05-12T13:00:19.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:19.518 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:19.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:19.519 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:19.612 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:19.613 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:19.613 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:19.613 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:19.613 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:19.614 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:19.614 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:19.614 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:19.614 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:19.614 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:19.615 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:19.615 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:19.615 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:19.615 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:19.616 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:19.616 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:19.616 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:19.616 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:19.616 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:19.617 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:19.617 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:19.617 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:19.617 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:19.618 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:19.618 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:19.618 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:19.618 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:19.618 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:19.619 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:19.619 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:19.620 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:19.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:19.722 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:19.723 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:19.723 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:19.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:19.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:19.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:19.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:20.189 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:20.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:20.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:20.189 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:20.672 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:20.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:20.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:20.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:20.673 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:20.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:20.674 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:20.674 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:20.674 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:21.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862920 2021-05-12T13:00:21.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862920 '!=' null 2021-05-12T13:00:21.158 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:21.159 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:21.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:21.159 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:21.252 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:21.253 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:21.253 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:21.253 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:21.253 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:21.254 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:21.254 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:21.254 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:21.255 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:21.255 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:21.255 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:21.255 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:21.256 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:21.256 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:21.256 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:21.256 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:21.256 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:21.257 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:21.257 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:21.257 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:21.258 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:21.258 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:21.258 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:21.258 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:21.259 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:21.259 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:21.259 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:21.259 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:21.260 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:21.260 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:21.260 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:21.362 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:21.363 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:21.363 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:21.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:21.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:21.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:21.364 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:21.365 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:21.723 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:21.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:21.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:21.724 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:22.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:22.220 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:22.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:22.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:22.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:22.221 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:22.222 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:22.222 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:22.222 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:22.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241963 2021-05-12T13:00:22.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241963 '!=' null 2021-05-12T13:00:22.783 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:22.784 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:22.784 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:22.784 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:22.877 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:22.878 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:22.878 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:22.878 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:22.879 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:22.879 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:22.879 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:22.879 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:22.879 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:22.880 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:22.880 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:22.880 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:22.880 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:22.881 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:22.881 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:22.881 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:22.881 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:22.881 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:22.882 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:22.882 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:22.882 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:22.882 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:22.882 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:22.883 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:22.883 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:22.883 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:22.883 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:22.884 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:22.884 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:22.884 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:22.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:22.987 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:22.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:22.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:22.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:22.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:22.988 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:22.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:22.989 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:23.357 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:23.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:23.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:23.357 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:23.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:23.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:23.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:23.857 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:23.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:23.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:23.858 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:23.858 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:23.859 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:24.349 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862985 2021-05-12T13:00:24.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862985 '!=' null 2021-05-12T13:00:24.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:24.350 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:24.350 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:24.351 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:24.443 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:24.443 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:24.444 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:24.444 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:24.444 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:24.444 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:24.445 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:24.445 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:24.445 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:24.446 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:24.446 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:24.446 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:24.446 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:24.447 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:24.447 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:24.447 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:24.447 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:24.448 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:24.448 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:24.448 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:24.448 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:24.449 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:24.449 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:24.449 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:24.450 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:24.450 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:24.450 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:24.450 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:24.451 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:24.451 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:24.451 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:24.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:24.554 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:24.554 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:24.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:24.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:24.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:24.555 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:24.556 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:24.908 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:24.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:24.909 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:24.910 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:25.404 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:25.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:25.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:25.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:25.405 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:25.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:25.406 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:25.406 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:25.406 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:25.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241992 2021-05-12T13:00:25.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241992 '!=' null 2021-05-12T13:00:25.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:25.886 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:25.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:25.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:25.979 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:25.979 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:25.980 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:25.980 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:25.980 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:25.980 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:25.980 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:25.981 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:25.981 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:25.982 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:25.982 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:25.982 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:25.982 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:25.983 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:25.983 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:25.983 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:25.983 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:25.984 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:25.984 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:25.984 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:25.984 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:25.985 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:25.985 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:25.985 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:25.985 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:25.985 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:25.986 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:25.986 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:25.986 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:25.986 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:25.988 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:26.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:26.091 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:26.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:26.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:26.092 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:26.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:26.093 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:26.094 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:26.459 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:26.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:26.459 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:26.460 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:27.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:27.008 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:27.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:27.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:27.009 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:27.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:27.010 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:27.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:27.010 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:27.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242022 2021-05-12T13:00:27.520 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242022 '!=' null 2021-05-12T13:00:27.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:27.521 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:27.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:27.521 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:27.615 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:27.616 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:27.616 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:27.616 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:27.617 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:27.617 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:27.617 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:27.617 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:27.618 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:27.618 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:27.618 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:27.618 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:27.618 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:27.619 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:27.619 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:27.619 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:27.619 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:27.620 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:27.620 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:27.620 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:27.620 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:27.621 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:27.621 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:27.621 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:27.621 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:27.621 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:27.622 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:27.622 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:27.622 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:27.622 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:27.624 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:27.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:27.726 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:27.726 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:27.726 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:27.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:27.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:27.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:27.727 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:28.082 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:28.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:28.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:28.083 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:28.568 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:28.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:28.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:28.569 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:28.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:28.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:28.570 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:28.570 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:28.571 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:29.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242022 2021-05-12T13:00:29.122 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242022 '!=' null 2021-05-12T13:00:29.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:29.123 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:29.124 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:29.124 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:29.217 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:29.217 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:29.217 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:29.218 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:29.218 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:29.219 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:29.219 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:29.219 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:29.220 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:29.220 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:29.220 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:29.221 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:29.221 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:29.221 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:29.222 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:29.222 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:29.222 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:29.223 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:29.223 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:29.224 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:29.224 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:29.224 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:29.225 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:29.225 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:29.226 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:29.226 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:29.227 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:29.227 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:29.227 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:29.228 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:29.228 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:29.327 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:29.328 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:29.328 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:29.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:29.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:29.329 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:29.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:29.330 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:29.692 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:29.692 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:29.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:29.693 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:30.173 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:30.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:30.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:30.174 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:30.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:30.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:30.175 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:30.175 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:30.175 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:30.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989332 2021-05-12T13:00:30.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989332 '!=' null 2021-05-12T13:00:30.719 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:30.720 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:30.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:30.720 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:30.812 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:30.812 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:30.813 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:30.813 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:30.813 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:30.814 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:30.814 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:30.814 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:30.814 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:30.814 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:30.815 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:30.815 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:30.815 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:30.815 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:30.816 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:30.816 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:30.816 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:30.816 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:30.816 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:30.817 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:30.817 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:30.817 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:30.817 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:30.817 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:30.818 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:30.818 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:30.818 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:30.818 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:30.818 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:30.819 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:30.821 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:30.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:30.922 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:30.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:30.923 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:30.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:30.924 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:30.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:30.925 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:31.279 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:31.279 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:31.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:31.280 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:31.768 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:31.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:31.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:31.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:31.769 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:31.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:31.770 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:31.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:31.770 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:32.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368322 2021-05-12T13:00:32.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368322 '!=' null 2021-05-12T13:00:32.322 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:32.323 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:32.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:32.323 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:32.416 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:32.417 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:32.417 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:32.417 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:32.417 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:32.418 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:32.418 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:32.418 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:32.418 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:32.419 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:32.419 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:32.419 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:32.419 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:32.419 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:32.420 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:32.420 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:32.420 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:32.421 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:32.421 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:32.421 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:32.421 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:32.421 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:32.422 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:32.422 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:32.422 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:32.422 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:32.423 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:32.423 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:32.423 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:32.423 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:32.425 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:32.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:32.527 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:32.527 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:32.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:32.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:32.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:32.528 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:32.529 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:32.885 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:32.885 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:32.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:32.886 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:33.371 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:33.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:33.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:33.372 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:33.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:33.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:33.373 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:33.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:33.374 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:33.862 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989341 2021-05-12T13:00:33.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989341 '!=' null 2021-05-12T13:00:33.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:33.863 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:33.864 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:33.864 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:33.956 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:33.957 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:33.957 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:33.957 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:33.958 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:33.958 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:33.958 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:33.958 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:33.959 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:33.959 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:33.959 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:33.959 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:33.960 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:33.960 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:33.960 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:33.960 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:33.961 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:33.961 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:33.961 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:33.961 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:33.962 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:33.962 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:33.962 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:33.962 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:33.963 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:33.963 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:33.963 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:33.963 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:33.964 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:33.964 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:33.965 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:34.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:34.067 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:34.067 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:34.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:34.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:34.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:34.068 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:34.069 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:34.423 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:34.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:34.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:34.424 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:34.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:34.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:34.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:34.917 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:34.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:34.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:34.918 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:34.918 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:34.919 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:35.402 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989341 2021-05-12T13:00:35.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989341 '!=' null 2021-05-12T13:00:35.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:35.403 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:35.403 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:35.404 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:35.497 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:35.498 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:35.498 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:35.498 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:35.498 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:35.499 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:35.499 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:35.499 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:35.499 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:35.500 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:35.500 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:35.500 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:35.500 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:35.500 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:35.501 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:35.501 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:35.501 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:35.501 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:35.501 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:35.502 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:35.502 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:35.502 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:35.502 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:35.503 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:35.503 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:35.503 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:35.503 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:35.503 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:35.504 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:35.504 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:35.506 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:35.607 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:35.608 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:35.608 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:35.608 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:35.608 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:35.608 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:35.609 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:35.609 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:35.971 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:35.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:35.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:35.972 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:36.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:36.466 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:36.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:36.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:36.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:36.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:36.467 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:36.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:36.468 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:36.960 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368326 2021-05-12T13:00:36.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368326 '!=' null 2021-05-12T13:00:36.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:36.961 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:36.961 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:36.962 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:37.052 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:37.052 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:37.052 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:37.052 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:37.053 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:37.053 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:37.053 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:37.053 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:37.054 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:37.054 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:37.054 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:37.054 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:37.055 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:37.055 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:37.055 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:37.055 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:37.056 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:37.056 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:37.056 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:37.056 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:37.057 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:37.057 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:37.057 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:37.057 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:37.059 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:37.060 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:37.161 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:37.162 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:37.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:37.162 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:37.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:37.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:37.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:37.163 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:37.524 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:37.524 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:37.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:37.525 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:38.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:38.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:38.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:38.012 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:38.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:38.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:38.013 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:38.013 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:38.014 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:38.494 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368329 2021-05-12T13:00:38.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368329 '!=' null 2021-05-12T13:00:38.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:38.495 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:38.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:38.495 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.190757/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:38.588 INFO:tasks.workunit.client.0.smithi191.stdout:{ 2021-05-12T13:00:38.589 INFO:tasks.workunit.client.0.smithi191.stdout: "local_reservations": { 2021-05-12T13:00:38.589 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:38.589 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:38.589 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:38.590 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [ 2021-05-12T13:00:38.590 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:38.590 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "1.0", 2021-05-12T13:00:38.590 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:38.591 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:38.591 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:38.591 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:38.591 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "3.0", 2021-05-12T13:00:38.591 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 190, 2021-05-12T13:00:38.592 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:38.592 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:38.592 INFO:tasks.workunit.client.0.smithi191.stdout: { 2021-05-12T13:00:38.592 INFO:tasks.workunit.client.0.smithi191.stdout: "item": "7.0", 2021-05-12T13:00:38.593 INFO:tasks.workunit.client.0.smithi191.stdout: "prio": 255, 2021-05-12T13:00:38.593 INFO:tasks.workunit.client.0.smithi191.stdout: "can_preempt": true 2021-05-12T13:00:38.593 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:38.593 INFO:tasks.workunit.client.0.smithi191.stdout: ] 2021-05-12T13:00:38.594 INFO:tasks.workunit.client.0.smithi191.stdout: }, 2021-05-12T13:00:38.594 INFO:tasks.workunit.client.0.smithi191.stdout: "remote_reservations": { 2021-05-12T13:00:38.594 INFO:tasks.workunit.client.0.smithi191.stdout: "max_allowed": 1000, 2021-05-12T13:00:38.595 INFO:tasks.workunit.client.0.smithi191.stdout: "min_priority": 0, 2021-05-12T13:00:38.595 INFO:tasks.workunit.client.0.smithi191.stdout: "queues": [], 2021-05-12T13:00:38.595 INFO:tasks.workunit.client.0.smithi191.stdout: "in_progress": [] 2021-05-12T13:00:38.595 INFO:tasks.workunit.client.0.smithi191.stdout: } 2021-05-12T13:00:38.596 INFO:tasks.workunit.client.0.smithi191.stdout:} 2021-05-12T13:00:38.597 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:38.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:38.699 INFO:tasks.workunit.client.0.smithi191.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:38.699 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:38.699 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:38.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:38.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:38.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:38.700 INFO:tasks.workunit.client.0.smithi191.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:38.783 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-05-12T13:00:38.786 DEBUG:teuthology.task.console_log:Killing console logger for smithi191 2021-05-12T13:00:38.787 DEBUG:teuthology.exit:Finished running handlers